{"id":"9a8b715e-844e-4f27-ab85-968b2518cd75","arxiv_id":"2412.11373","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"In a full-rule Codenames benchmark, nine LLMs show distinct play styles, generalize better across teammates than word-vector agents, and still lose to self-matched word-vector agents on raw score.","lead":"This paper turns the word game Codenames into a test for large language models, pitting several models against each other as clue-givers and guessers. It finds the models have distinct risk-taking styles and cooperate more flexibly with unfamiliar teammates than older word-vector programs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central 'more generalisable' claim is not directly tested: LLM teams are only paired with other LLMs and word-vector teams only with other word-vector agents, so the wider-range conclusion rests on a missing cross-type comparison.","rationale":"The reader's weakest_assumption was that rule approximations (substring-based no-derived-words check and lenient retry policy) could distort rankings. That is a real validity threat, but the more fundamental gap for the central claim is the missing cross-type teammate comparison. The abstract claims LLMs are 'more generalisable to a wider range of teammates than prior techniques,' yet the experiments only compare LLMs with LLMs and word-vector agents with word-vector agents. The word-vector failure with LLM clues is reported, but the reverse pairing (LLM guesser with word-vector codemaster) was not run, despite being listed as future work. This means the headline claim is not directly evidenced. I still support the paper conditionally: the benchmark, code, and within-class comparisons are useful, and adding the cross-type experiment or softening the claim would resolve the issue. The reader's own rationale also mentions that the 'more generalisable' claim is broader than the tested conditions, which is why I mark partial agreement rather than full disagreement. I do not see a reason to change the CONDITIONAL verdict, but the condition should explicitly include either running the missing cross-type teams or revising the abstract and Section I claims to the tested scope.","tokens_in":20941,"tokens_out":7708,"duration_ms":72178,"concrete_test":"Run the single-team Codenames protocol with each of the four LLMs (GPT-4o, Gemini-1.5, Sonnet-3.5, Llama-3.1) as guesser paired with Word2Vec, GloVe, and Combined as codemaster, and vice versa where vocabulary constraints allow, using the same 100 seeds and scoring. If the mean scores of these cross-type teams are comparable to the LLM-LLM and word-vector-word-vector baselines (within, say, 2 points), the generalization claim is supported; if they degrade substantially (e.g., by more than 3 points) or fail due to uninterpretable clues, the 'wider range' claim must be weakened to 'range of LLM teammates.'","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that 'LLM agents are more generalisable to a wider range of teammates than prior techniques' (Abstract; Section I). What would have to be true is that LLMs maintain performance across a broader set of teammate types than word-vector agents do. The experiments do not test this. Section IV.A.2 states that word-vector agents 'are unable to interpret any words which are not present in their training set,' that the authors 'were unable to produce reliable performance results for games with both LLM and Word-Vector agents on the same team,' and that they 'have instead chosen to evaluate each agent group separately.' Consequently, all LLM-teammate pairs are LLM-LLM and all word-vector pairs are word-vector-word-vector; the teammate sets are disjoint. The observed robustness of LLM cross-pairs relative to, e.g., Word2Vec-GloVe could reflect within-class compatibility (all LLMs share natural-language semantics and a common prompt format) rather than a genuinely larger range of teammates. The paper itself defers the decisive condition to future work: Section VII notes that word-vector agents 'can still operate as Codemaster alongside an LLM guesser,' acknowledging that this pairing was not run. Without LLM-word-vector (or human) teammate pairs, the 'wider range' claim is broader than the evidence. This is a scope gap, not an internal inconsistency, but it is load-bearing because the abstract's headline contribution rests on it.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Codenames as a benchmark for evaluating large language models, extending an existing Codenames AI framework to support the full rules and two-team play. It evaluates nine LLMs (o1-preview, o1-mini, o3-mini, GPT-4o, Gemini-1.5, Sonnet-3.5, DeepSeek-R1, DeepSeek-V3, Llama-3.1) and three word-vector agents (Word2Vec, GloVe, Combined) over 100 trials of single-team and two-team games, reporting scores, win-rates, and playstyle metrics. The authors claim that LLM agents are more generalisable to a wider range of teammates than prior word-vector techniques, that each LLM exhibits a distinct emergent playstyle, and that o1-preview performs best in the single-team setting. Code, prompts, outputs, and results are made publicly available.","tokens_in":21195,"tokens_out":7872,"duration_ms":63274,"significance":"Assuming the central claims survive scrutiny, Codenames is a well-motivated and behaviorally rich benchmark for LLM language understanding, theory of mind, and cooperation. The paper's strengths include a publicly available framework with full rules, 100 trials per condition, detailed per-pair statistics, and transparent reporting of API costs. The playstyle metrics (clue number, stop-early/stop-late rates) are a useful contribution and could support future work on emergent LLM behaviour. However, the headline generalization claim is broader than the evidence, and several methodological choices need tightening before the benchmark results can be taken at face value.","major_comments":[{"comment":"The central claim that 'LLM agents are more generalisable to a wider range of teammates than prior techniques' is not directly tested. In the experiments, all LLM teams are paired only with other LLMs and all word-vector teams only with other word-vector agents; Section IV.A.2 states that the authors 'were unable to produce reliable performance results for games with both LLM and Word-Vector agents on the same team' and therefore evaluate each agent group separately. Section VII then notes that word-vector agents 'can still operate as Codemaster alongside an LLM guesser,' acknowledging that a cross-type pairing was feasible but not run. As a result, the observed robustness of LLM cross-pairs could reflect within-class compatibility (shared natural-language semantics and prompt format) rather than a genuinely wider range of teammates. Please either run the missing cross-type conditions (e.g., word-vector codemaster with LLM guesser) or narrow the claim to generalization across LLM teammates.","section":"Abstract; Section I; Section IV.A.2; Section VII"},{"comment":"The model-selection procedure weakens the breadth of the claims. Six of the fifteen initially selected LLMs are excluded after 'preliminary testing' because they could not consistently follow the rules, but no results, criteria, or error rates are reported for them; this makes the comparison susceptible to selection bias and makes it difficult to interpret 'LLMs' as a class. In addition, five of the nine retained models (o1-preview, o1-mini, o3-mini, DeepSeek-R1, DeepSeek-V3) are evaluated only in self-pair configurations (Section IV.D.1), so the evidence for cross-teammate generalization actually rests on four models. Please report the excluded models' performance in an appendix, define a principled inclusion rule, or explicitly frame the results as applying only to high-performing, format-compliant LLMs.","section":"Section IV.A.1; Section IV.D.1"},{"comment":"Win-rate and mean-score differences are reported without confidence intervals or significance tests. With 100 trials per condition, a 52% vs. 48% win-rate is within binomial sampling error, so statements such as Sonnet-3.5 having 'the highest average win-rate for the red team of 56.75%' and GPT-4o having 'the highest overall win-rate for the blue team of 58.75%' are not supported without uncertainty quantification. The same applies to the mean-score rankings in Table I. Please add confidence intervals or significance tests (e.g., bootstrap or exact binomial tests) for the key comparisons, especially those underlying the playstyle and generalization claims.","section":"Table II; Section VI.A.2; Table I"},{"comment":"The rule approximations can systematically bias the comparisons. The invalid-clue checker is a substring match, and Section VI.B.2 gives a concrete example where the valid clue Education is rejected because it contains the board word CAT; combined with the lenient retry policy and the random fallback after ten failures, this can favor models that produce short, substring-safe, format-compliant clues over models that reason about meaning. Because the paper does not report per-model invalid-response frequencies or re-run results with a more accurate adjudicator, it is unclear how much of the observed playstyle and generalization differences is an artifact of the checker. Please quantify the frequency of invalid responses per model and test sensitivity with an improved checker or human adjudication.","section":"Section II.C; Section VI.B.2"}],"minor_comments":[{"comment":"The 'GloVe vs. Combined' row reports win-rates of 63% and 35%, which sum to 98%; please correct the typo or explain any draws, since Codenames normally has no draw outcome.","section":"Table II"},{"comment":"There is a typo in the guesser prompting description: 'invliad response' should be 'invalid response'.","section":"Section IV.C.2"},{"comment":"The column header 'Civillian avg' should be spelled 'Civilian avg'.","section":"Table I"},{"comment":"The Spearman rank correlation of 0.821 between average clue number and loss percentage is reported without a p-value or a statement of the sample size; please clarify what is being correlated and how many model pairs are included.","section":"Section VI.A.1"},{"comment":"The text says the figure confirms 'our previous ordering of models from most risky to most cautious' but then lists only Sonnet-3.5, Gemini-1.5, GPT-4o, and Llama-3.1; please make explicit that this is a subset of the full ordering given in Section VI.A.1.","section":"Figure 2"},{"comment":"References [28] and [41] appear to cite the same Sidji, Smith, and Rogerson paper on human-AI collaboration in Codenames; please merge or disambiguate them.","section":"References [28] and [41]"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable fit for the journal, and the authors have been transparent about costs and limitations. The main concern is that the abstract and conclusion overclaim relative to the missing cross-type comparison; if the authors can add the decisive experiments or substantially qualify the claims, I would support publication. I do not see issues with citation practice or novelty disclosure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Steve — quick take. This paper is worth engaging with, but its headline claim runs ahead of the evidence. What is genuinely new is the artifact: a full two-team Codenames implementation under official rules, with LLM agent support, plus a public repo with prompts, outputs, and raw results. That is real, citable infrastructure. The evaluation spans nine state-of-the-art LLMs, and the behavioral analysis—riskiness via clue numbers, stop early/late rates, role-specific performance—is the most valuable part. The o1-preview result and the inversion between single-team and two-team scoring are interesting and worth a second look.\n\nThe soft spots are real but mostly addressable. The biggest is the generalization claim. The abstract says LLM agents are “more generalisable to a wider range of teammates than prior techniques,” but no LLM-word-vector team or human baseline appears anywhere. Word-vector agents play only with word-vector teammates; LLMs play only with LLMs. Section IV.A.2 says the mixed pairing failed because word-vector guessers cannot interpret LLM clues, and Section VII defers the feasible mixed case (LLM guesser plus word-vector codemaster) to future work. So the “wider range” conclusion is a scope gap, and it is load-bearing, since it is the paper’s headline. The authors should soften it or run the missing comparison.\n\nSecond, six of fifteen models were dropped after preliminary testing showed poor rule adherence. That is defensible, and they document it, but it means the “nine LLMs” results are conditional on a rule-following filter. The failure modes of the six are worth reporting as a result in their own right. Third, the 100-trial win rates come with no confidence intervals or significance tests; several differences in Table II (e.g., 52% vs 48%) are almost certainly noise. Fourth, the substring invalid-clue checker rejects valid clues—Education contains CAT—and the lenient retry plus random fallback could systematically favor short, format-safe clues. These are acknowledged, but the paper never quantifies how much they could distort model rankings.\n\nTo be fair, the paper is unusually honest about its limitations and ships reproducible code. The comparative playstyle observations are plausible. But the central “more generalisable” claim needs either evidence or a rewrite.\n\nBottom line: this deserves a serious referee. A revision that fixes the overclaim, adds uncertainty bounds, and reports the filtered models’ failures would make it a solid benchmark paper. I’d bring it to a reading group, and I’d cite the framework if I were building on Codenames.","headline":"Useful benchmark artifact with a headline claim that runs ahead of the evidence: the cross-family generalization result is asserted but never actually tested.","tokens_in":21724,"tokens_out":3068,"would_cite":true,"duration_ms":27572,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLMs handle a wider range of Codenames teammates than word-vector agents, though they don't yet beat those agents when paired with an identical partner.","keywords":["Codenames","Large Language Models","AI benchmark","theory of mind","cooperative game playing","emergent playstyle","word embeddings","epistemic reasoning"],"falsifier":"Run the single-team and two-team experiments with a morphological 'no derived words' checker that accepts Education as valid (instead of rejecting it for containing CAT) and without the retry/random-fallback mechanism; if the model rankings and win-rates change materially, then the paper's central claims depend on the framework's rule approximations rather than on LLM reasoning ability.","tokens_in":20702,"feed_emoji":"🧩","tokens_out":10473,"duration_ms":78696,"temperature":0.7,"pith_summary":"This paper proposes the word-association board game Codenames as a benchmark for evaluating large language models, arguing that the game's need for language understanding, theory of mind, and epistemic reasoning makes it a better test than static knowledge quizzes. The authors compare nine LLMs (including GPT-4o, Claude 3.5 Sonnet, Gemini 1.5, and Llama 3.1) against traditional word-vector agents in both a single-team scoring version and a full two-team competitive version of the game. Their central result is that current LLMs do not beat the word-vector agents when paired with an identical technique, but they lose far less when paired with a different agent, which the authors read as greater generalisability to unknown teammates. Each LLM also displayed a distinct, stable risk profile, and OpenAI's o1-preview achieved the best single-team score despite a risky style. The paper offers this as evidence that Codenames can serve as a behaviourally rich LLM benchmark.","feed_headline":"LLMs cope with any Codenames teammate; word-vector bots don't","feed_subtitle":"A full-rules Codenames benchmark shows LLMs generalize across teammates, offering a new way to test theory of mind.","key_machinery":"The central object is the Codenames game itself, used as a two-role benchmark in which one agent (codemaster) gives a single-word clue plus a number, and a partner (guesser) selects words from a 25-word board. The paper's key mechanism is the measurement of cross-agent generalisation: it pairs every codemaster with every guesser and records mean score, win-rate, loss rate, average clue number, and stop-early/stop-late rates. The framework updates a prior single-team version to full two-team rules, which changes the optimal risk strategy and is itself part of the claim about benchmark design.","core_discovery":"The paper's central claim is that LLM agents are more generalisable to a wider range of teammates than prior word-vector techniques, and that each LLM exhibits a unique emergent style of play. The quantitative evidence shows that mixed-model LLM pairs (e.g., GPT-4o with Gemini-1.5, or Llama-3.1 with Sonnet-3.5) perform roughly as well as same-model pairs in the single-team version, whereas word-vector pairs such as Word2Vec with GloVe see their mean score jump from roughly 5-6 to 12 and their loss rate from 0% to over 20%. In the two-team version, GloVe beats Word2Vec by a 72% win-rate when they face off, while LLM pairings mostly stay within a 40-60 win-rate band. The single-team results put o1-preview ahead of all other LLMs (mean score 8.41, with the next-best at 9.70 and most other LLM pairs above 10), and qualitative observations report LLMs drawing on cultural references like 'Hogwarts' to connect three board words at once, something no word-vector agent could do. The authors conclude that the full rules of Codenames expose cooperation and reasoning traits that simpler benchmarks miss.","pith_inferences":["A testable extension of this paper would replace the substring-only 'no derived words' check with a morphological or dictionary-based checker; if models like Gemini-1.5 and Llama-3.1, which tend to give longer meaning-rich clues, improve their ranks relative to models that favour short safe clues, then the reported playstyle ordering is partly an artifact of the current checker.","The two-team result suggests a general design principle for LLM game benchmarks: the payoff function (mean score vs win-rate) selects the optimal risk attitude, so benchmark authors should report both scoring and win-rate to avoid conflating strategic caution with inability.","The paper's observation that LLMs produce culturally-grounded clues such as 'Hogwarts' points to a concrete way to test shared-knowledge theory of mind: measure how well human guessers, not just LLM guessers, solve clues from each model, and compare against word-vector clues."],"forward_implications":["If LLMs generalise across teammates, Codenames becomes a reproducible way to test theory of mind in models: a good guesser must infer what clue the codemaster intended, and a good codemaster must anticipate how a different teammate will interpret a clue.","Because each model has a stable playstyle, single-model leaderboard scores are only meaningful relative to a fixed teammate; future benchmark reporting should include the full teammate matrix rather than just self-play results.","The reversal between game versions—cautious play wins the single-team scoring game, risky play wins the two-team win-rate game—shows that benchmark design decisions (scoring vs win-rate, presence of an opponent) determine which model appears best, so conclusions about 'better' LLMs must be tied to the rule set.","The o1-preview result suggests that reasoning-specialised models may belong on a separate evaluation track from chat models for cooperative language games.","Word-vector agents remain the best same-model baseline (0% loss, scores near 5-7), so they should stay in the benchmark as the point of comparison for LLM generalisation rather than being removed."],"supporting_citations":[{"why":"Codenames rulebook; defines the official game rules the benchmark replicates.","marker":"[13]"},{"why":"Prior Codenames AI competition; provides the simplified single-team framework this paper extends to full rules.","marker":"[14]"},{"why":"Original word-vector Codenames agents; the finding that they fail with differing teammates is the baseline this paper challenges.","marker":"[15]"},{"why":"A transformer-based Codenames agent that outperformed word-vector approaches; acts as the neural predecessor to LLM agents.","marker":"[30]"},{"why":"ThinkSum prompting technique that improved LLM guesser performance; one of the prior LLM benchmarks on Codenames.","marker":"[34]"},{"why":"Prior comparison of prompt-engineering methods on GPT-4-1106; establishes that prompting changes playstyle but not overall performance, motivating the model-level comparison here.","marker":"[26]"},{"why":"Pre-trained Word2Vec vectors; used to build one of the word-vector baseline agents.","marker":"[38]"},{"why":"GloVe embeddings; used to build another word-vector baseline agent.","marker":"[39]"},{"why":"Concatenated word-vector approach; used as the third baseline agent.","marker":"[40]"}],"fun_headline_variants":["Codenames: LLMs team up with anyone, word-vector bots fail","LLMs are flexible Codenames partners; Word2Vec and GloVe are not","Codenames benchmark: LLMs beat word-vector agents at teamwork","Mixed LLM pairs match same-model pairs in Codenames; old bots tank","LLM agents show unique Codenames skills, outperform word vectors"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the framework's approximations of official Codenames rules—the substring-only check for the 'no derived words' rule, the lenient retry policy for invalid responses, and the random fallback after ten failures—do not distort which models rank higher; if these approximations favour models that produce short, substring-safe, format-compliant clues rather than models that reason about meaning, the comparative conclusions about generalisability and playstyle would not hold.","fun_headline_variants_meta":{"raw":{"variants":["Codenames: LLMs team up with anyone, word-vector bots fail","LLMs are flexible Codenames partners; Word2Vec and GloVe are not","Codenames benchmark: LLMs beat word-vector agents at teamwork","Mixed LLM pairs match same-model pairs in Codenames; old bots tank","LLM agents show unique Codenames skills, outperform word vectors"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000657,"raw_usage":{"total_tokens":3044,"prompt_tokens":1021,"completion_tokens":2023,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":637,"completion_tokens_details":{"reasoning_tokens":1920}},"tokens_in":637,"tokens_out":2023,"duration_ms":13564,"temperature":1.0,"reasoning_tokens":1920,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T14:59:44.222378+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the single-team and two-team experiments with a morphological 'no derived words' checker that accepts Education as valid (instead of rejecting it for containing CAT) and without the retry/random-fallback mechanism; if the model rankings and win-rates change materially, then the paper's central claims depend on the framework's rule approximations rather than on LLM reasoning ability.","supporting_citations":[{"cited_title":"Chv ´atil, Codenames","cited_arxiv_id":null,"evidence_quote":"Codenames rulebook; defines the official game rules the benchmark replicates."},{"cited_title":"The codenames ai competition,","cited_arxiv_id":null,"evidence_quote":"Prior Codenames AI competition; provides the simplified single-team framework this paper extends to full rules."},{"cited_title":"Cooperation and Codenames: Understanding Natural Language Processing via Code- names,","cited_arxiv_id":null,"evidence_quote":"Original word-vector Codenames agents; the finding that they fail with differing teammates is the baseline this paper challenges."},{"cited_title":"Word autobots: Using transformers for word association in the game codenames,","cited_arxiv_id":null,"evidence_quote":"A transformer-based Codenames agent that outperformed word-vector approaches; acts as the neural predecessor to LLM agents."},{"cited_title":"ThinkSum: Probabilis- tic reasoning over sets using large language models,","cited_arxiv_id":null,"evidence_quote":"ThinkSum prompting technique that improved LLM guesser performance; one of the prior LLM benchmarks on Codenames."},{"cited_title":"Prompt Engineering ChatGPT for Co- denames,","cited_arxiv_id":null,"evidence_quote":"Prior comparison of prompt-engineering methods on GPT-4-1106; establishes that prompting changes playstyle but not overall performance, motivating the model-level comparison here."},{"cited_title":"Concatenated power mean word embeddings as universal cross-lingual sentence representations,","cited_arxiv_id":null,"evidence_quote":"Concatenated word-vector approach; used as the third baseline agent."}],"review_version":1}