{"id":"dacfe26e-31ba-4f70-bbde-164105394561","arxiv_id":"2411.10422","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A Balderdash simulation framework shows LLMs generate plausible fake definitions but fail to reason over game rules or adapt strategy, with the effect strongest on rare words.","lead":"The authors built a multi-agent simulation where large language models play Balderdash, inventing definitions for obscure words and voting on the true definition. A smart generalist would read it to see a proposed benchmark for measuring creativity, deception, and strategic reasoning in AI systems, especially their fragility with rare vocabulary.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The Llama judge is the sole measurement instrument for both word familiarity and all game metrics, but it is validated only on 80 GPT-generated examples (F1 0.74) and never on actual game outputs; the negative conclusions may be an artifact of judge false negatives.","rationale":"The reader identified the same load-bearing concern: the Llama judge is used as the known-word labeler, the in-game judge, and a player, while its only external validation is F1 = 0.74 on 80 GPT-generated examples. That is exactly the weakest point of the paper's empirical argument. The paper's strongest claim — that LLMs are unfamiliar with most Balderdash words and fail to reason about rules or converge on dominant strategies — depends on judge decisions at every step. A false-negative rate of roughly 26% on true definitions is enough to make convergence impossible to detect under Eq. 6 and to deflate the 'Known' word counts in Table 2. The paper itself flags the missing validation in its Limitations section, which strengthens the concern rather than resolving it. A human-labeling study on actual game outputs, stratified by model and word frequency, would settle whether the negative results survive. Because the paper proposes a reusable framework and includes prompts, the appropriate verdict remains CONDITIONAL: the framework and negative findings are plausible but not yet verified. The reader's verdict is unchanged.","tokens_in":14952,"tokens_out":5103,"duration_ms":53392,"concrete_test":"Have human annotators label a stratified sample of actual game outputs — e.g., 200 judge decisions drawn from the Leaderboard and Convergence experiments, covering all five models, both word decks, and both judge verdicts — for whether each generated definition is semantically equivalent to the reference definition and whether it matches at least one dictionary meaning. Compare the Llama judge's verdicts against these human labels, computing per-model F1 and the false-negative rate on true definitions. Then recompute LKR, TDR, and the convergence curves in Eq. 6 using human-corrected labels. If the corrected LKR approaches 1 for models on their 'Known' word subsets, or if corrected convergence appears, the paper's claim that models show no strategy convergence is an artifact of judge noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central negative result — 'LLMs are not familiar with more than half of the Balderdash words and ... none ... showed signs of correct reasoning based on game rules or strategy convergence' — rests entirely on the Llama semantic-equivalence judge. That judge is used in three roles: (i) to construct the 'Known Balderdash Words' subsets in Section 4.2.1 by majority vote over five definitions, (ii) to compute 'judge decision' and 'llm knows one' for every round in Section 5, and (iii) as a player in the leaderboard experiment, creating a potential self-enhancement loop. The only external validation is F1 = 0.74 on 80 GPT-generated examples (Section 4.3, Table 3); there are no human labels on actual game outputs, no per-model error rates, and no estimate of Llama's false-negative rate on rare words. This matters quantitatively. Since the reported recall is 0.74, the judge labels roughly 26% of true definitions as false. The convergence criterion in Eq. 6 requires LKR_n > 1 - epsilon; unless epsilon is set above 0.26, even a model that always writes the correct definition cannot be observed to converge. Similarly, a 26% false-negative rate would directly depress the 'Known' word counts in Table 2, so the claim that models are 'not familiar with more than half of the Balderdash words' conflates lexical familiarity with one-shot, high-temperature definition generation judged by an unvalidated noisy classifier. The Limitations section itself acknowledges that the judge should be evaluated on definitions generated by all LLMs, not only GPT, and that self-enhancement bias should be checked. Until that validation is done on the actual game outputs, the paper's headline conclusions are not empirically grounded.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-agent simulation framework for the game Balderdash, in which several LLMs generate definitions for rare words, a judge LLM decides semantic equivalence to dictionary definitions, and players vote on which definition is correct. The authors define five round-level metrics (TDR, LKR, DR, CGR, AS) and run three experiments: a leaderboard over history types and word decks, a convergence study on model-specific \"Known\" word sets, and a game-rules experiment comparing 0 vs 50 points for writing the true definition. The central claims are that LLMs handle frequent words better than rare words, that they do not converge to the dominant strategy even when given full history, and that they fail to reason over game rules and adapt to incentive changes. The framework and code are open-sourced on GitHub.","tokens_in":15274,"tokens_out":4169,"duration_ms":44422,"significance":"If the central findings hold, the paper provides a reusable, structured environment for probing creativity, deception, and strategic adaptation in LLM agents, filling a gap left by games such as Avalon and Werewolf that focus on deception and reasoning but not on creative definition generation. The framework's strengths include its centralized game engine, the use of multiple open-source models, explicit prompts and history formats, and the release of code and datasets. However, the validity of every quantitative claim rests on the accuracy of the LLM judge, which is validated on only 80 GPT-generated examples with an F1 of 0.74 and is never checked against human labels on actual game outputs. Because the judge is also used to construct the Known-word test sets and is one of the players, there is a partial circularity. The qualitative finding about poor performance on infrequent words is plausible and consistent with prior work on frequency bias, but the magnitude and the negative conclusions about strategy adaptation need stronger measurement support before they can be accepted.","major_comments":[{"comment":"The judge is the sole measurement instrument for all game outcomes, yet its validation is limited to F1=0.74 on 80 GPT-generated examples. No human labels are collected for the actual game outputs, and no per-model or per-word-type error rates are reported. Since TDR, LKR, the Known-word subsets (Table 2), and the convergence criterion (Eq. 6) all derive from this judge, a false-negative rate of roughly 26% can directly deflate the \"Known\" word counts and the observed LKR, making the conclusion that \"LLMs are not familiar with more than half of the Balderdash words\" conflate lexical familiarity with one-shot, high-temperature generation judged by an unvalidated noisy classifier. The authors must supply a human evaluation on a sample of game outputs, per-model judge accuracy, and a sensitivity analysis showing how the main results change under alternative judge decisions or thresholds.","section":"§4.3, Table 3; §5.1; §6.2"},{"comment":"The convergence definition LKR_n > 1 - epsilon, for all n > T, is incompatible with the reported judge recall of 0.74. Even a model that always generates a correct definition cannot be observed to converge if the judge randomly labels 26% of true definitions as false, unless epsilon is set above that error rate. The paper never states the value of epsilon used in the experiments, so the statement that \"none of the models converge\" is not informative. The authors should either specify epsilon and compare convergence against a judge-noise-corrected ceiling, or report convergence as a statistical comparison to the maximum achievable LKR under the measured judge error.","section":"§5.1, Eq. (6); §6.2"},{"comment":"The Known-word subsets are constructed using Llama as a semantic-equivalence judge over five high-temperature generations per word, and the same Llama judge is then used to measure LKR and TDR in the convergence and game-rules experiments. This creates a partial circularity: a word is \"known\" for a model only if Llama already accepted that model's definition, so LKR on Known words can be inflated for models whose outputs align with Llama's judgments, while failures of other models may reflect judge mismatch rather than lack of lexical knowledge. The authors acknowledge self-enhancement bias in the Limitations section, but they do not address the fact that the judge is also the arbiter of the test-set construction. A human-labeled validation on actual game outputs for each player model is needed to break this loop.","section":"§4.2.1; §4.3; §6.2; §6.3"},{"comment":"The claim that there is no improvement or trend over rounds is based on visual inspection of plots whose standard deviations are down-scaled by a factor of 0.2 for presentation. No statistical test for convergence is reported, despite five runs per setting. Given the visible fluctuations and the non-trivial error bars, the authors should report a quantitative trend test (e.g., linear mixed-effects model, repeated-measures ANOVA, or a test of the slope of LKR over rounds) to support the conclusion of non-adaptation.","section":"§6.2, Figure 1"},{"comment":"The game-rules experiment is described as being run with one player, but the scoring rules and the voting phase involve other players' definitions and votes. The manuscript does not clarify how the voting phase is simulated in this single-player setup, nor does it explain how the reported standard deviations are computed. This makes it difficult to interpret the small differences between the 0-point and 50-point conditions, especially since the LKR and TDR values are still judge-dependent. The authors should specify the exact game configuration for this experiment and, ideally, include results from multi-player runs as a robustness check.","section":"§6.3, Table 6"}],"minor_comments":[{"comment":"The sentence \"theDasher (the leader of each round)\" contains a spacing typo; it should read \"the Dasher\".","section":"§3"},{"comment":"The prompts contain typos: \"V ote\" instead of \"Vote\" in the voting prompt, \"definiton\" instead of \"definition\" in the history column description, and \"correct_definiton\" instead of \"correct_definition\" in the CSV column description.","section":"Appendix D, Figures 8 and 10"},{"comment":"The parameter epsilon is introduced in the convergence definition but never assigned a value in the experiments. Please state the chosen value and, if multiple values were tested, report the sensitivity of the convergence results to epsilon.","section":"§5.1, Eq. (6)"},{"comment":"The text says that \"Mistral has the most wins\" on the Basic Frequent English Words dataset, but the tables do not define or count \"wins.\" Please either define a win criterion (e.g., highest AS or highest rank in each game) and report the counts, or revise the claim to match the presented metrics.","section":"§6.1, Tables 4 and 5"},{"comment":"The \"Basic Frequent English Words\" dataset is described as containing the most frequent English words, but its average NGRAM frequency of 6.3e-5 is still very low in absolute terms. Reporting the distribution or a comparison with the Balderdash word frequencies would help readers calibrate the difference.","section":"§4.2.2"},{"comment":"It would be helpful to state explicitly that the 80 data points come from 40 words, each with one correct and one deceiving GPT-generated definition, and to report the judge's accuracy separately for correct and deceiving definitions, as false negatives and false positives may affect different metrics.","section":"§4.3, Table 3"}],"recommendation":"major_revision","confidential_remarks":"The concerns about the LLM judge are serious but fixable within the scope of the manuscript: adding a human evaluation on game outputs, reporting per-model judge accuracy, specifying epsilon, and adding statistical tests would substantially strengthen the claims. The circularity of the Known-word construction is also a load-bearing issue, but it can be addressed by validating the judge on the actual outputs of each model. I see no sign of dishonest reporting; the Limitations section already anticipates several of these issues, which speaks well of the authors' awareness. The framework itself is a useful contribution to LLM-agent evaluation, so I would not recommend rejection; major revision seems appropriate given that the current measurement validation is insufficient for the strength of the negative conclusions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Let me give you the short version: the Balderdash simulation is a real addition to the LLM-agent evaluation toolbox. It gives the community a game that separates generating creative fake definitions from detecting the true one, and it comes with a centralized engine, complete prompts, and a GitHub repo. That’s more than most benchmark papers ship. The empirical finding that performance collapses on infrequent vocabulary, and that models do not adapt their strategy when the scoring rules change, is worth taking seriously.\n\nThe soft spot is exactly where the stress-test note points. Every headline metric runs through Llama as the semantic-equivalence judge. That judge was selected on 80 GPT-generated examples with F1 0.74 (recall 0.74, precision 0.74). It is then used to build the “Known Balderdash Words” subsets, to label every generated definition in the leaderboard and convergence experiments, and it also serves as a player. A 26% false-negative rate means the known-word lists are likely too conservative, and the convergence criterion LKR > 1 − epsilon is structurally hard to satisfy for any epsilon below about 0.26. So the claim that LLMs are “not familiar with more than half of the Balderdash words” is not well grounded: it conflates lexical familiarity with one-shot high-temperature generation judged by a noisy classifier. The paper actually acknowledges some of this in the Limitations section, which is to its credit, but the acknowledgment does not fix the load-bearing nature of the judge.\n\nThere are smaller issues: no significance tests on the comparative claims, and the standard deviations overlap in many rows, so the leaderboard rankings should be read as descriptive. The free parameters (epsilon, history window, temperature) are not specified for the convergence experiment, which makes reproduction harder.\n\nOn the whole, the paper is honest and the framework is reusable. The right next step is human-labeling a sample of actual game outputs from all models, reporting per-model judge error, and re-estimating the key metrics. That is doable, and would turn this from a promising preprint into a solid benchmark. I would send it to peer review, with the clear message that the judge validation is the gate.","headline":"A useful new Balderdash benchmark, but the central negative results are built on a single weakly validated LLM judge.","tokens_in":15900,"tokens_out":1998,"would_cite":false,"duration_ms":18518,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that large language models playing Balderdash fail to reason about game rules or learn from history, especially on rare words, and provides a simulation framework that makes these failures measurable.","keywords":["Balderdash","multi-agent simulation","LLM creativity","LLM deception","LLM-as-judge","rare vocabulary","strategy convergence","game-based evaluation"],"falsifier":"Take a random sample of the game transcripts, have human annotators label whether each player's definition is semantically equivalent to the dictionary definition, and compare those labels with the Llama judge's decisions; if human-judge agreement is substantially lower than 0.74 F1, the paper's headline failure rates become measurement artifacts, and if agreement is high, the failure rates are real model limitations.","tokens_in":14726,"feed_emoji":"🎲","tokens_out":7690,"duration_ms":67287,"temperature":0.7,"pith_summary":"This paper builds a multi-agent simulation of the word game Balderdash—where players invent plausible definitions for obscure words and vote on which definition is the true one—and uses it to test whether large language models can be creative, deceptive, and strategically adaptive. The central claim is that they largely cannot: the models know fewer than half of the rare Balderdash words, vote poorly, and do not converge on the highest-scoring strategy even when the scoring rules and a history of past rounds are supplied. The failure is more pronounced for infrequent vocabulary than for common words, suggesting that rare vocabulary degrades not only word knowledge but also rule-based reasoning. The framework contributes a reproducible testbed with defined metrics and an LLM judge for semantic equivalence, so the disappointing results can be measured, compared, and revisited.","feed_headline":"LLM agents lose at Balderdash on rare words","feed_subtitle":"A Balderdash simulation shows LLMs fail to adapt even when history and rules make the best move obvious.","key_machinery":"The load-bearing mechanism is the centralized Balderdash game engine: it fixes the rules and scoring, rotates the Dasher/judge role, tracks rounds in a database, and feeds each player a prompt conditioned on game rules and, optionally, a CSV history of previous rounds. The judge is a separate LLM—Llama, selected by comparing five LLMs against 80 human-labeled examples—that decides whether a player's definition is semantically equivalent to the dictionary definition; this judge decision is the basis for the True Definition Ratio and LLM Knows Ratio metrics. The other core pieces are the word decks (225 rare Balderdash words versus 2,865 frequent English words), the per-model known-word subsets, and the five round-level metrics—True Definition Ratio, LLM Knows Ratio, Deception Ratio, Correct Guess Ratio, and Average Score—that turn raw game transcripts into comparable behavior. The design isolates the effect of word frequency by running identical games on rare and frequent decks, and isolates strategy learning by varying history information (none, mini, full) and the points awarded for a correct definition.","core_discovery":"The paper's discovery, stated on its own terms, is that LLM agents playing Balderdash do not demonstrate the creativity-plus-reasoning balance the game demands. Across five models, none of them became familiar with more than half of the Balderdash words, none performed reliably in the voting phase, and none showed correct reasoning based on game rules or strategy convergence derived from historical context. When the optimal strategy was unambiguous—for example, writing the true definition when it earns fifty points—the models still did not shift their behavior accordingly, and with zero points for a correct definition they continued writing true definitions and thereby scored nothing. The authors interpret the pattern as evidence that infrequent vocabulary in the input makes LLMs more susceptible to reasoning failure, a vulnerability their framework makes visible and quantifiable.","pith_inferences":["A testable extension implied by the convergence failure: if players received explicit post-round feedback stating which strategy would have maximized points, rather than only a CSV of outcomes, some models might converge, which would localize the deficit in credit assignment rather than in rule comprehension.","Because the judge was validated only on GPT-generated definitions, the reported True Definition Ratio and LLM Knows Ratio may confound model performance with judge error; re-labeling a sample of actual game outputs with human annotators would separate the two.","The framework's rare-word decks could double as a hallucination probe: a model that invents confident, plausible-but-wrong definitions for obscure words is performing the same operation as a hallucinating system, and Balderdash gives it a reward signal for exactly that behavior."],"forward_implications":["LLM agents cannot be assumed to adapt their behavior to simple incentive changes; even clear scoring rules and round-by-round feedback leave their strategies essentially unchanged.","Rare vocabulary is a second axis of failure: models that perform credibly on frequent English words degrade sharply on obscure words in both generation and voting, so benchmarks that sample only common language will overstate agent competence.","Game-based evaluation with an LLM judge can expose failures that static creativity tests miss, because Balderdash couples generation of plausible fake definitions with inference about the true definition in one interaction.","The leaderboard results are frequency-dependent: Mistral looked strongest on frequent words while Phi looked strongest on rare words, so claims about which model is best at deception or detection are only meaningful relative to the word distribution tested."],"supporting_citations":[{"why":"Establishes the frequency bias that motivates the rare-word decks and the interpretation that infrequent vocabulary destabilizes LLM reasoning.","marker":"Kang and Choi (2023)"},{"why":"Supplies the definition-modeling method used to decide which Balderdash words each model knows.","marker":"Jhirad et al. (2023)"},{"why":"Provides the LLM-as-judge paradigm the paper adopts for semantic-equivalence evaluation.","marker":"Zheng et al. (2024)"},{"why":"Source of the Oxford 3000 frequent-word list used to build the Basic Frequent English Words deck.","marker":"Oxford University Press (2024)"},{"why":"Dictionary API that supplies reference definitions for the frequent-word deck.","marker":"Merriam-Webster (2024)"},{"why":"NGRAMS dataset used to compute the average word frequencies that distinguish the rare and frequent decks.","marker":"Trenkmann (2023)"},{"why":"BERTScore is tested and rejected as an alternative judge, narrowing the design to an LLM judge.","marker":"Song et al. (2021)"},{"why":"Llama 3 model card; Llama is the model chosen as judge and one of the game participants.","marker":"AI@Meta (2024)"},{"why":"GPT-3.5 API model used as a participant and as the generator of the judge-validation dataset.","marker":"OpenAI (2024)"}],"fun_headline_variants":["LLMs fail Balderdash on rare words","Rare words stump LLM agents in Balderdash","LLM agents miss obvious Balderdash strategy","Balderdash exposes LLM reasoning blind spot","Infrequent words break LLM game reasoning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results depend on the assumption that the Llama judge correctly decides semantic equivalence for every model and every word, yet that judge was validated on only 80 GPT-generated examples with an F1 of 0.74 and never checked against human labels on actual game outputs.","fun_headline_variants_meta":{"raw":{"variants":["LLMs fail Balderdash on rare words","Rare words stump LLM agents in Balderdash","LLM agents miss obvious Balderdash strategy","Balderdash exposes LLM reasoning blind spot","Infrequent words break LLM game reasoning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000256,"raw_usage":{"total_tokens":1560,"prompt_tokens":914,"completion_tokens":646,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":530,"completion_tokens_details":{"reasoning_tokens":570}},"tokens_in":530,"tokens_out":646,"duration_ms":5746,"temperature":1.0,"reasoning_tokens":570,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T19:37:50.596816+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the game transcripts, have human annotators label whether each player's definition is semantically equivalent to the dictionary definition, and compare those labels with the Llama judge's decisions; if human-judge agreement is substantially lower than 0.74 F1, the paper's headline failure rates become measurement artifacts, and if agreement is high, the failure rates are real model limitations.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the definition-modeling method used to decide which Balderdash words each model knows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of the Oxford 3000 frequent-word list used to build the Basic Frequent English Words deck."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Dictionary API that supplies reference definitions for the frequent-word deck."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NGRAMS dataset used to compute the average word frequencies that distinguish the rare and frequent decks."}],"review_version":1}