{"id":"3b871679-e137-4a60-9c08-a086e5d64066","arxiv_id":"2506.21490","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"AH2AC2 provides an open, reproducible Hanabi benchmark for human-AI ad-hoc coordination, with withheld proxy agents and baselines showing a large gap to human-level play.","lead":"This paper introduces AH2AC2, a benchmark for testing whether AI agents can cooperate with human-like Hanabi playing partners, built around a small open dataset of human games and API-hosted human proxy agents whose weights stay secret. It reports baseline scores showing that zero-shot coordination methods currently outperform methods that try to learn from limited human data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The proxy agents' human-likeness is validated only against the same H-group distribution; without direct human-AI play, leaderboard scores are not established as measures of human-AI coordination.","rationale":"The central claim is that HDR-IPPO proxies serve as human-like evaluation partners, and the paper itself defers direct human-AI play to future work in Section 7. All three validation strands in Section 5 draw on the same H-group hanab.live population used for training: BC cross-play, held-out action prediction, and aggregate behavioral metrics. The KL-regularized RL step drifts from the human action distribution (Table 3), so even the internal evidence is not entirely one-sided. However, this is not an internal inconsistency or a reproducibility failure; the paper is transparent about the limitation, and the open dataset, code, controlled API, and baseline suite are independently valuable contributions. My proposed human-subject study would directly test whether proxy-based scores and rankings predict human-based scores and rankings; without such a test, the leaderboard should be interpreted as measuring coordination with H-group-derived proxies, not with humans generally. This reinforces, rather than changes, the reader's CONDITIONAL verdict.","tokens_in":32416,"tokens_out":5815,"duration_ms":72915,"concrete_test":"Preregister a human-subject evaluation in which 20 players familiar with H-group conventions and 20 players unfamiliar with them each play a balanced set of games (e.g., 30 games) with the two 2-player human proxies and with a human partner of similar skill, while a fixed set of candidate agents (e.g., OBL, HDR-IPPO, IPPO) is evaluated both with the proxies and with the human partners. Compare mean and median scores and score distributions using a pre-specified equivalence margin (e.g., 1.5 points) and test whether the rank ordering of candidate agents is preserved between proxy-based and human-based evaluation. If proxy-based and human-based rankings diverge or scores differ beyond the margin, the central claim that the proxies are human-like evaluation partners is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5 validates the human proxies with three strands of evidence: cross-play against BC policies trained on the same H-group dataset, action prediction on held-out games from that same population, and aggregate IPP/Communicativeness statistics. None of these establish the load-bearing property of the central claim, namely that a candidate agent's score with a proxy predicts its score with a human partner. The cross-play matrices in Figure 2 compare policies drawn from one convention distribution, so high compatibility shows only that the proxy did not drift from the BC training signal; it says nothing about how an unfamiliar human would react to the proxy's play. Table 3 shows that the HDR-IPPO refinement consistently moves the action distribution away from the human data (accuracy deltas -0.03 to -0.08, loss increases +0.05 to +0.08), so even the internal evidence for human-likeness is not one-sided. The paper explicitly defers direct human-AI play to future work in Section 7. Therefore, the claim that the proxies are 'human-like evaluation partners' is currently an assumption rather than a demonstrated property. This does not invalidate the benchmark artifact, but it means leaderboard rankings cannot yet be read as human-AI coordination performance for humans outside the H-group population.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AH2AC2, a benchmark for ad-hoc human-AI coordination in Hanabi, and proposes human proxy agents as cheap, reproducible stand-ins for human partners. The proxies are trained by behavioral cloning on 147,621 games from the hanab.live H-group population, then refined with IPPO regularized by a KL term toward the BC policy. The authors open-source a limited dataset of 3,079 games, provide a controlled evaluation API and leaderboard, and report baselines including IPPO, BC, BR-BC, OBL, OP, FCP, and a prompted DeepSeek-R1 agent. The paper also defines an optional action-prediction challenge. Section 5 validates the proxies via self-play scores, cross-play with BC policies, action-prediction metrics on held-out games, and two behavioral statistics (IPP and Communicativeness); Section 7 explicitly defers direct human-AI play validation to future work.","tokens_in":32662,"tokens_out":4111,"duration_ms":45533,"significance":"If the central claim holds, AH2AC2 would be a valuable community resource: it is the first open-source human Hanabi dataset, it provides a controlled evaluation protocol, and it assembles a broad set of baselines that clarify the difficulty of human-AI coordination. The HDR-IPPO recipe, the hosted API, and the open baselines are concrete and reproducible contributions, and the paper gives credit where prior datasets were closed. However, the paper's headline claim is that the proxies are 'human-like evaluation partners', and the evidence for this is indirect: all validation strands are measured against the same H-group population used for training. The leaderboard therefore measures coordination with a learned proxy, not yet with humans. The benchmark artifact is useful even under that caveat, but the claims and the 'golden standard' label in Table 5 outrun the current evidence.","major_comments":[{"comment":"The load-bearing property of the central claim—that scores achieved with the proxies measure human-AI coordination—is not demonstrated. The three validation strands all measure agreement with the same H-group population used for training: action-prediction loss/accuracy on held-out games from that population, cross-play with BC policies trained on that population, and two aggregate behavioral statistics. None of these establishes that a candidate's score with a proxy predicts its score with a human, especially a human outside H-group conventions. Moreover, Table 3 shows that the HDR-IPPO refinement systematically degrades the direct human-likeness metrics relative to BC (accuracy −0.03 to −0.08; loss +0.05 to +0.08). The paper itself states in Section 7 that 'The ultimate validation of our human proxy agents requires direct human-AI play' and defers it. The Abstract and Section 4.2 claims that the proxies 'serve as robust, cheap, and reproducible human-like evaluation partners' therefore outrun the current evidence.","section":"Section 5.2, Table 3, Section 7"},{"comment":"The leaderboard footnote labels the human proxies as a 'golden standard' because they are not constrained by game limits. This is circular relative to the paper's own caveat: the proxies are the proposed substitute for human evaluation, so their scores cannot serve as the criterion against which candidate agents are judged until the proxy-human correspondence is established. I recommend either adding a human-subject validation study or relabeling these scores as 'proxy-coordination scores' and removing 'golden standard' until such validation exists.","section":"Table 5, Section 4.3"},{"comment":"The evaluation protocol is described inconsistently: the main text says registered participants receive access to 1,000 evaluation games, while Appendix A.7 states that the evaluation is 2,000 games total (1,000 in each of two- and three-player settings). Please disambiguate. Additionally, the text calls pre-registration 'the gold standard for empirical science'; pre-registration alone does not prevent iterative fitting to the hosted proxies across repeated evaluation requests, so the integrity guarantee needs a concrete statement about how many evaluation runs a participant may request and what controls exist.","section":"Section 4.3, Appendix A.7"}],"minor_comments":[{"comment":"The abstract says the proxies are trained on 'a large-scale human dataset', but the full dataset is closed and only 3,079 games are open-sourced; please make this asymmetry explicit in the abstract.","section":"Abstract and Section 4.1"},{"comment":"There is a typo: 'gives them access to to 1,000 evaluation games' should read 'access to 1,000 evaluation games'.","section":"Section 4.1"},{"comment":"In the Dec-POMDP definition, the sentence 'i.e. sH is always a terminal state' lacks a period and is missing the preceding comma; please fix the punctuation.","section":"Section 2"},{"comment":"The phrase 'highlight the brittleness of BC policies in unfamiliar scenarios' and 'BC policies' poor single-player performance' use 'single-player' where 'self-play' is the intended term; please align the terminology with the rest of the paper.","section":"Section 5.2"},{"comment":"The qualitative analysis of the proxy's convention-following is based on a single game; please state explicitly that this is illustrative rather than a systematic behavioral evaluation.","section":"Appendix A.6"},{"comment":"The LLM prompt templates contain several typos that may affect reproducibility, including 'your shop' for 'your chop', 'rnak clue' for 'rank clue', 'follwing' for 'following', 'Teamamte' for 'Teammate', and 'has not been discarded yet, where discarding'.","section":"Appendix A.10"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern lands: the proxies' human-likeness is validated only against the training population, and the paper explicitly defers direct human-AI play to future work. This is a load-bearing gap for the paper's central claim, but it is fixable within the manuscript's scope by adding a human-subject study or by substantially reframing the claims and leaderboard interpretation. I would not recommend rejection, because the benchmark artifact and open-source contributions are solid and useful even with a more modest interpretation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi — quick take on the AH2AC2 paper. The benchmark itself is the real contribution: an open limited dataset, API-hosted hidden proxies, pre-registration, and a leaderboard with baselines including the first FCP run in Hanabi and DeepSeek-R1. That's genuinely useful infrastructure, and the code and data are available. The method (BC then KL-regularized IPPO) is inherited from Bakhtin, Hu, Jacob, and Cornelisse, and they cite those works properly.\n\nThe soft spot is exactly where the stress-test lands. The claim that the proxies are 'human-like evaluation partners' rests on evidence that is almost entirely in-distribution. Cross-play with BC policies trained on the same H-group data, action prediction on a held-out slice of the same population, and IPP/communicativeness aggregates show the proxies stayed close to the training distribution. They do not show that a candidate's score with a proxy predicts its score with a human. Table 3 even shows the HDR-IPPO refinement moves action distributions away from human data (accuracy down 0.03–0.08, loss up 0.05–0.08), so the internal evidence is not one-sided. The authors acknowledge in Section 7 that direct human-AI play is the ultimate validation, but that missing piece is the load-bearing part of the central claim.\n\nI wouldn't sink the paper over this. The benchmark's value does not hinge on proxies being indistinguishable from all humans; it is explicitly scoped to H-group hanab.live players, and that scope is stated. What is not yet supported is the broader statement that leaderboard scores measure human-AI coordination generally. That should be framed as a limitation rather than an achieved property.\n\nMinor issues: the leaderboard has no error bars or variance; DeepSeek-R1 is only 100 games; FCP uses a single seed. All minor and fixable.\n\nOverall, a useful and honest paper with a clear, addressable weakness in validation. I would like to see it in the literature after there is at least an engagement with the correlation between proxy scores and human scores. It deserves a real peer review, and I would cite it if I worked on Hanabi or coordination benchmarks.","headline":"Useful benchmark infrastructure, but the proxies' human-likeness is not yet established; treat leaderboard scores as coordination with these specific policies, not with humans.","tokens_in":33237,"tokens_out":4081,"would_cite":true,"duration_ms":41366,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Human proxies make Hanabi AI benchmarking cheap and reproducible","keywords":["human-AI coordination","ad-hoc teamplay","Hanabi","human proxy agents","behavioural cloning","regularised reinforcement learning","zero-shot coordination","benchmark"],"falsifier":"A direct human-AI play experiment in which human participants play Hanabi with both the human proxies and with other humans, comparing scores and subjective coordination quality: if proxy-AI teams score systematically lower than human-AI teams (or proxy-human teams differ from human-human teams), the proxies do not stand in for real human partners.","tokens_in":32232,"feed_emoji":"🃏","tokens_out":3212,"duration_ms":29780,"temperature":0.7,"pith_summary":"The paper is trying to establish that expensive, hard-to-reproduce human evaluation can be replaced by human proxy agents—policies trained on a large corpus of human Hanabi games that behave like the humans who supplied the data. If the proxies genuinely stand in for human partners, then AI coordination research gains a standardized, cheap, and reproducible evaluation protocol, which the authors deliver as the Ad-Hoc Human-AI Coordination Challenge (AH2AC2). The paper further claims that combining behavioural cloning with KL-regularized independent PPO yields proxies that are both stronger players than pure imitation and still human-like, and it provides baselines showing that current zero-shot coordination methods, human-data methods, population-based methods, and LLM agents all struggle on this benchmark.","feed_headline":"Human proxies make Hanabi AI benchmarking cheap and reproducible","feed_subtitle":"Policies trained on 147,000 human games stand in for costly human partners in a new coordination challenge.","key_machinery":"The central object is the human proxy agent trained with Human-Data-Regularised IPPO (HDR-IPPO). The procedure first trains an LSTM-based behavioural cloning policy on a large-scale human dataset (101,096 two-player and 46,525 three-player games from hanab.live following H-group conventions), then initializes from the best BC checkpoint and continues with IPPO while adding a KL regularisation term anchoring the policy to the BC action distribution, weighing the IPPO objective against KL divergence with a weight lambda.","core_discovery":"The central claim is that human proxy agents trained by behavioural cloning on a large-scale human dataset followed by KL-regularized independent PPO serve as robust, cheap, and reproducible human-like evaluation partners for ad-hoc human-AI coordination in Hanabi. The paper demonstrates that these proxies substantially improve self-play scores over pure BC policies—especially in three-player games, where zero-score games drop from over 70% to well under 1%—while retaining human-like behaviour, evidenced by cross-play compatibility with BC policies, comparable action-prediction accuracy on held-out human games, and nearly identical information-per-play and communicativeness metrics. On the resulting leaderboard, no baseline matches the human proxies themselves, and OBL, which uses no human data, outperforms methods that use the provided limited human data, highlighting the difficulty of efficient human-data use.","pith_inferences":["The proxies' validation is internal to one community (H-group on hanab.live); the paper itself defers direct human-play validation. If the goal is coordination with general humans, the proxies may over-represent a specific convention system, and leaderboard scores might not transfer to casual or differently conventioned players.","A direct testable extension would be a human study comparing human scores and subjective experience when paired with the proxies versus with real humans; if scores diverge, the proxies' role as a surrogate would need recalibration.","The KL-regularisation mechanism is quite general: the same BC-plus-regularised-RL recipe could create human proxies for other cooperative games or real-world human-AI collaboration settings where human demonstrations are available but live evaluation is expensive.","The near-identical behavioural metrics (IPP and communicativeness) between proxies and the human dataset are coarse summaries; finer-grained behavioural distributions (e.g., convention-following rates per game position) could reveal whether the proxies reproduce human strategy spread or only its average."],"forward_implications":["AI coordination research gains a standardized, low-cost evaluation protocol for human-AI coordination in Hanabi, with 1,000-game evaluation runs through a controlled API instead of costly human studies.","The benchmark can track progress in data-efficient human-AI coordination, since participants receive only 3,079 open-source games while the proxies are trained on the full closed dataset.","The baseline results set a reference: OBL's strong zero-shot performance without human data suggests that existing methods underutilize limited human data, and the lagging performance of BC, BR-BC, FCP, OP, IPPO, and DeepSeek-R1 defines the current gap.","The action-prediction sub-challenge provides a complementary, replicable measure of human-likeness via teacher-forced cross-entropy loss on held-out human games.","If the proxies are accepted as human stand-ins, the leaderboard provides a meaningful ranking of human-AI coordination ability that can guide future algorithmic development."],"supporting_citations":[{"why":"Supplies the motivation that self-play agents overfit to conventions and the BR-BC method of best-responding to a behavioural cloning policy, which the paper adapts as a baseline.","marker":"(Carroll et al., 2019)"},{"why":"These works provide the empirical basis that KL-regularised RL keeps policies close to human play styles, which HDR-IPPO builds upon.","marker":"(Hu et al., 2022; Bakhtin et al., 2022; Cornelisse & Vinitsky, 2024)"},{"why":"Off-Belief Learning is the strongest zero-shot coordination baseline in the leaderboard and provides the main comparison point for human-data-free methods.","marker":"(Hu et al., 2021)"},{"why":"Fictitious Co-Play is the population-based baseline that the paper evaluates for the first time in Hanabi.","marker":"(Strouse et al., 2021)"},{"why":"Supplies the behavioural metrics (Information per Play and Communicativeness) used to validate that proxy behaviour matches the human dataset.","marker":"(Canaan et al., 2020)"},{"why":"Provides the Independent PPO method that HDR-IPPO uses as the RL backbone.","marker":"(de Witt et al., 2020)"},{"why":"The IPPO configuration and environment infrastructure that the paper adopts for baselines.","marker":"(Rutherford et al., 2023)"},{"why":"Provides the DeepSeek-R1 model used to benchmark off-the-shelf LLM capabilities for human-AI coordination.","marker":"(DeepSeek-AI et al., 2025)"}],"fun_headline_variants":["Hanabi proxy agents replace costly human testing","AI coordination benchmark now runs on human proxies","Cheap human proxies expose AI coordination gaps in Hanabi","No baseline beats human proxies in new Hanabi challenge","Human-like proxies make Hanabi AI evaluation reproducible"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that human proxy agents, trained on and validated against H-group hanab.live players, faithfully represent the humans an AI agent will actually coordinate with.","fun_headline_variants_meta":{"raw":{"variants":["Hanabi proxy agents replace costly human testing","AI coordination benchmark now runs on human proxies","Cheap human proxies expose AI coordination gaps in Hanabi","No baseline beats human proxies in new Hanabi challenge","Human-like proxies make Hanabi AI evaluation reproducible"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000183,"raw_usage":{"total_tokens":1308,"prompt_tokens":930,"completion_tokens":378,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":546,"completion_tokens_details":{"reasoning_tokens":306}},"tokens_in":546,"tokens_out":378,"duration_ms":4302,"temperature":1.0,"reasoning_tokens":306,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:23:25.074635+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct human-AI play experiment in which human participants play Hanabi with both the human proxies and with other humans, comparing scores and subjective coordination quality: if proxy-AI teams score systematically lower than human-AI teams (or proxy-human teams differ from human-human teams), the proxies do not stand in for real human partners.","supporting_citations":[],"review_version":1}