Evaluating Creativity and Deception in Large Language Models: A Simulation Framework for Multi-Agent Balderdash

Alireza S. Ziabari; Elnaz Rahmati; Jesse Thomason; Morteza Dehghani; Parsa Hejabi; Preni Golazizian

REVIEW 5 major objections 6 minor 37 references

Evaluating Creativity and Deception in Large Language Models: A Simulation Framework for Multi-Agent Balderdash

T0 review · 5 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that large language models playing Balderdash fail to reason about game rules or learn from history, especially on rare words, and provides a simulation framework that makes these failures measurable.

desk verdict A useful new Balderdash benchmark, but the central negative results are built on a single weakly validated LLM judge. read the letter →

arxiv 2411.10422 v1 pith:PAFWGQAP submitted 2024-11-15 cs.MA cs.AI

Parsa Hejabi , Elnaz Rahmati , Alireza S. Ziabari , Preni Golazizian , Jesse Thomason , Morteza Dehghani This is my paper · ORCID

classification cs.MAcs.AI

keywords Balderdashmulti-agentsimulationLLMcreativitydeceptionLLM-as-judgerarevocabularystrategyconvergencegame-basedevaluation

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper builds a multi-agent simulation of the word game Balderdash—where players invent plausible definitions for obscure words and vote on which definition is the true one—and uses it to test whether large language models can be creative, deceptive, and strategically adaptive. The central claim is that they largely cannot: the models know fewer than half of the rare Balderdash words, vote poorly, and do not converge on the highest-scoring strategy even when the scoring rules and a history of past rounds are supplied. The failure is more pronounced for infrequent vocabulary than for common words, suggesting that rare vocabulary degrades not only word knowledge but also rule-based reasoning. The framework contributes a reproducible testbed with defined metrics and an LLM judge for semantic equivalence, so the disappointing results can be measured, compared, and revisited.

What carries the argument

The load-bearing mechanism is the centralized Balderdash game engine: it fixes the rules and scoring, rotates the Dasher/judge role, tracks rounds in a database, and feeds each player a prompt conditioned on game rules and, optionally, a CSV history of previous rounds. The judge is a separate LLM—Llama, selected by comparing five LLMs against 80 human-labeled examples—that decides whether a player's definition is semantically equivalent to the dictionary definition; this judge decision is the basis for the True Definition Ratio and LLM Knows Ratio metrics. The other core pieces are the word decks (225 rare Balderdash words versus 2,865 frequent English words), the per-model known-word subsets, and the five round-level metrics—True Definition Ratio, LLM Knows Ratio, Deception Ratio, Correct Guess Ratio, and Average Score—that turn raw game transcripts into comparable behavior. The design isolates the effect of word frequency by running identical games on rare and frequent decks, and isolates strategy learning by varying history information (none, mini, full) and the points awarded for a correct definition.

What would settle it

Take a random sample of the game transcripts, have human annotators label whether each player's definition is semantically equivalent to the dictionary definition, and compare those labels with the Llama judge's decisions; if human-judge agreement is substantially lower than 0.74 F1, the paper's headline failure rates become measurement artifacts, and if agreement is high, the failure rates are real model limitations.

Watch

Extended reading notes

Core claim

The paper's discovery, stated on its own terms, is that LLM agents playing Balderdash do not demonstrate the creativity-plus-reasoning balance the game demands. Across five models, none of them became familiar with more than half of the Balderdash words, none performed reliably in the voting phase, and none showed correct reasoning based on game rules or strategy convergence derived from historical context. When the optimal strategy was unambiguous—for example, writing the true definition when it earns fifty points—the models still did not shift their behavior accordingly, and with zero points for a correct definition they continued writing true definitions and thereby scored nothing. The authors interpret the pattern as evidence that infrequent vocabulary in the input makes LLMs more susceptible to reasoning failure, a vulnerability their framework makes visible and quantifiable.

Load-bearing premise

The results depend on the assumption that the Llama judge correctly decides semantic equivalence for every model and every word, yet that judge was validated on only 80 GPT-generated examples with an F1 of 0.74 and never checked against human labels on actual game outputs.

Editorial extensions

If this is right

LLM agents cannot be assumed to adapt their behavior to simple incentive changes; even clear scoring rules and round-by-round feedback leave their strategies essentially unchanged.
Rare vocabulary is a second axis of failure: models that perform credibly on frequent English words degrade sharply on obscure words in both generation and voting, so benchmarks that sample only common language will overstate agent competence.
Game-based evaluation with an LLM judge can expose failures that static creativity tests miss, because Balderdash couples generation of plausible fake definitions with inference about the true definition in one interaction.
The leaderboard results are frequency-dependent: Mistral looked strongest on frequent words while Phi looked strongest on rare words, so claims about which model is best at deception or detection are only meaningful relative to the word distribution tested.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A testable extension implied by the convergence failure: if players received explicit post-round feedback stating which strategy would have maximized points, rather than only a CSV of outcomes, some models might converge, which would localize the deficit in credit assignment rather than in rule comprehension.
Because the judge was validated only on GPT-generated definitions, the reported True Definition Ratio and LLM Knows Ratio may confound model performance with judge error; re-labeling a sample of actual game outputs with human annotators would separate the two.
The framework's rare-word decks could double as a hallucination probe: a model that invents confident, plausible-but-wrong definitions for obscure words is performing the same operation as a hallucinating system, and Balderdash gives it a reward signal for exactly that behavior.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Let me give you the short version: the Balderdash simulation is a real addition to the LLM-agent evaluation toolbox. It gives the community a game that separates generating creative fake definitions from detecting the true one, and it comes with a centralized engine, complete prompts, and a GitHub repo. That’s more than most benchmark papers ship. The empirical finding that performance collapses on infrequent vocabulary, and that models do not adapt their strategy when the scoring rules change, is worth taking seriously.

The soft spot is exactly where the stress-test note points. Every headline metric runs through Llama as the semantic-equivalence judge. That judge was selected on 80 GPT-generated examples with F1 0.74 (recall 0.74, precision 0.74). It is then used to build the “Known Balderdash Words” subsets, to label every generated definition in the leaderboard and convergence experiments, and it also serves as a player. A 26% false-negative rate means the known-word lists are likely too conservative, and the convergence criterion LKR > 1 − epsilon is structurally hard to satisfy for any epsilon below about 0.26. So the claim that LLMs are “not familiar with more than half of the Balderdash words” is not well grounded: it conflates lexical familiarity with one-shot high-temperature generation judged by a noisy classifier. The paper actually acknowledges some of this in the Limitations section, which is to its credit, but the acknowledgment does not fix the load-bearing nature of the judge.

There are smaller issues: no significance tests on the comparative claims, and the standard deviations overlap in many rows, so the leaderboard rankings should be read as descriptive. The free parameters (epsilon, history window, temperature) are not specified for the convergence experiment, which makes reproduction harder.

On the whole, the paper is honest and the framework is reusable. The right next step is human-labeling a sample of actual game outputs from all models, reporting per-model judge error, and re-estimating the key metrics. That is doable, and would turn this from a promising preprint into a solid benchmark. I would send it to peer review, with the clear message that the judge validation is the gate.

Referee Report

5 major / 6 minor

Summary. The paper proposes a multi-agent simulation framework for the game Balderdash, in which several LLMs generate definitions for rare words, a judge LLM decides semantic equivalence to dictionary definitions, and players vote on which definition is correct. The authors define five round-level metrics (TDR, LKR, DR, CGR, AS) and run three experiments: a leaderboard over history types and word decks, a convergence study on model-specific "Known" word sets, and a game-rules experiment comparing 0 vs 50 points for writing the true definition. The central claims are that LLMs handle frequent words better than rare words, that they do not converge to the dominant strategy even when given full history, and that they fail to reason over game rules and adapt to incentive changes. The framework and code are open-sourced on GitHub.

Significance. If the central findings hold, the paper provides a reusable, structured environment for probing creativity, deception, and strategic adaptation in LLM agents, filling a gap left by games such as Avalon and Werewolf that focus on deception and reasoning but not on creative definition generation. The framework's strengths include its centralized game engine, the use of multiple open-source models, explicit prompts and history formats, and the release of code and datasets. However, the validity of every quantitative claim rests on the accuracy of the LLM judge, which is validated on only 80 GPT-generated examples with an F1 of 0.74 and is never checked against human labels on actual game outputs. Because the judge is also used to construct the Known-word test sets and is one of the players, there is a partial circularity. The qualitative finding about poor performance on infrequent words is plausible and consistent with prior work on frequency bias, but the magnitude and the negative conclusions about strategy adaptation need stronger measurement support before they can be accepted.

major comments (5)

[§4.3, Table 3; §5.1; §6.2] The judge is the sole measurement instrument for all game outcomes, yet its validation is limited to F1=0.74 on 80 GPT-generated examples. No human labels are collected for the actual game outputs, and no per-model or per-word-type error rates are reported. Since TDR, LKR, the Known-word subsets (Table 2), and the convergence criterion (Eq. 6) all derive from this judge, a false-negative rate of roughly 26% can directly deflate the "Known" word counts and the observed LKR, making the conclusion that "LLMs are not familiar with more than half of the Balderdash words" conflate lexical familiarity with one-shot, high-temperature generation judged by an unvalidated noisy classifier. The authors must supply a human evaluation on a sample of game outputs, per-model judge accuracy, and a sensitivity analysis showing how the main results change under alternative judge decisions or thresholds.
[§5.1, Eq. (6); §6.2] The convergence definition LKR_n > 1 - epsilon, for all n > T, is incompatible with the reported judge recall of 0.74. Even a model that always generates a correct definition cannot be observed to converge if the judge randomly labels 26% of true definitions as false, unless epsilon is set above that error rate. The paper never states the value of epsilon used in the experiments, so the statement that "none of the models converge" is not informative. The authors should either specify epsilon and compare convergence against a judge-noise-corrected ceiling, or report convergence as a statistical comparison to the maximum achievable LKR under the measured judge error.
[§4.2.1; §4.3; §6.2; §6.3] The Known-word subsets are constructed using Llama as a semantic-equivalence judge over five high-temperature generations per word, and the same Llama judge is then used to measure LKR and TDR in the convergence and game-rules experiments. This creates a partial circularity: a word is "known" for a model only if Llama already accepted that model's definition, so LKR on Known words can be inflated for models whose outputs align with Llama's judgments, while failures of other models may reflect judge mismatch rather than lack of lexical knowledge. The authors acknowledge self-enhancement bias in the Limitations section, but they do not address the fact that the judge is also the arbiter of the test-set construction. A human-labeled validation on actual game outputs for each player model is needed to break this loop.
[§6.2, Figure 1] The claim that there is no improvement or trend over rounds is based on visual inspection of plots whose standard deviations are down-scaled by a factor of 0.2 for presentation. No statistical test for convergence is reported, despite five runs per setting. Given the visible fluctuations and the non-trivial error bars, the authors should report a quantitative trend test (e.g., linear mixed-effects model, repeated-measures ANOVA, or a test of the slope of LKR over rounds) to support the conclusion of non-adaptation.
[§6.3, Table 6] The game-rules experiment is described as being run with one player, but the scoring rules and the voting phase involve other players' definitions and votes. The manuscript does not clarify how the voting phase is simulated in this single-player setup, nor does it explain how the reported standard deviations are computed. This makes it difficult to interpret the small differences between the 0-point and 50-point conditions, especially since the LKR and TDR values are still judge-dependent. The authors should specify the exact game configuration for this experiment and, ideally, include results from multi-player runs as a robustness check.

minor comments (6)

[§3] The sentence "theDasher (the leader of each round)" contains a spacing typo; it should read "the Dasher".
[Appendix D, Figures 8 and 10] The prompts contain typos: "V ote" instead of "Vote" in the voting prompt, "definiton" instead of "definition" in the history column description, and "correct_definiton" instead of "correct_definition" in the CSV column description.
[§5.1, Eq. (6)] The parameter epsilon is introduced in the convergence definition but never assigned a value in the experiments. Please state the chosen value and, if multiple values were tested, report the sensitivity of the convergence results to epsilon.
[§6.1, Tables 4 and 5] The text says that "Mistral has the most wins" on the Basic Frequent English Words dataset, but the tables do not define or count "wins." Please either define a win criterion (e.g., highest AS or highest rank in each game) and report the counts, or revise the claim to match the presented metrics.
[§4.2.2] The "Basic Frequent English Words" dataset is described as containing the most frequent English words, but its average NGRAM frequency of 6.3e-5 is still very low in absolute terms. Reporting the distribution or a comparison with the Balderdash word frequencies would help readers calibrate the difference.
[§4.3, Table 3] It would be helpful to state explicitly that the 80 data points come from 40 words, each with one correct and one deceiving GPT-generated definition, and to report the judge's accuracy separately for correct and deceiving definitions, as false negatives and false positives may affect different metrics.

Circularity Check

0 steps flagged · score 0.0 of 10

No definitional circularity; the LLM-judge measurement loop is a validity concern, not a circular derivation.

full rationale

The paper's derivation chain is empirical rather than definitional. The central negative result ('LLMs are not familiar with more than half of the Balderdash words and ... none ... showed signs of correct reasoning') is an interpretation of measured game outcomes, not an equation that reduces to its inputs. The Llama judge is selected against 80 human-labeled examples (Section 4.3, Table 3), which provides independent grounding for the measurement instrument; the judge is not a parameter fitted to the quantity it later evaluates. Using the same judge to construct the Known-word sets (Section 4.2.1) and to compute LKR/TDR (Section 5.1) creates a plausible measurement-validity threat—false negatives would depress observed LKR and known-word counts—and the paper itself acknowledges this in the Limitations section ('this might lead to false negatives') and the possibility of self-enhancement bias. But this is a bias/error concern, not a circular reduction: LKR and TDR are outputs of a game process judged by a fixed, externally benchmarked classifier, and they are not equal to the judge's labels by construction. The only self-citation (Singh et al., 2024) is background related-work and is not load-bearing. No uniqueness theorem, no ansatz smuggled via citation, and no renaming of a known result are present. The correctness risk from judge error should be assessed separately; it does not make the derivation circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No free parameters are fitted to optimize a loss; the hand-set experimental constants (temperature, majority threshold, epsilon, history window) are listed because they directly shape the reported results. The central claims rest on domain assumptions about the judge, the probing procedure, and the validity of game metrics as proxies for creativity and reasoning.

free parameters (4)

Known-word majority threshold = 3 of 5
A word is labeled as known when at least three of five high-temperature outputs are judged semantically equivalent by Llama; this threshold determines the datasets for the convergence and game-rules experiments.
Inference temperature = 0.9
Used for all definition generation to increase diversity; the paper notes that high temperature may cause false negatives for rare words.
Convergence epsilon = not specified
Equation (6) defines convergence as LKR > 1 minus epsilon for all rounds after T, but epsilon and T are never instantiated, so the convergence claim is not crisply testable.
History window size = not specified
The framework gives a moving window of previous rounds, but the paper does not state the window length, so history-condition results cannot be reproduced exactly.

assumptions (4)

domain assumption The LLM judge's semantic-equivalence decisions approximate human judgments well enough to be used as ground truth for all metrics.
Invoked in Section 4.3 and Section 5.1; the only external validation is F1 0.74 on 80 GPT-generated examples.
ad hoc to paper Probing each model five times at temperature 0.9 and taking a Llama-judged majority identifies the words the model knows.
Introduced in Section 4.2.1 to construct Known datasets; the paper itself notes this may produce false negatives.
domain assumption Game-performance metrics (LKR, TDR, DR, CGR, AS) are valid proxies for creativity, deception, and logical reasoning.
Section 5.1 defines the metrics and Section 6 interprets them as evidence about reasoning and strategy.
domain assumption Word frequency from NGRAMS is a valid proxy for how familiar a model is with a word.
Used in Section 4.2 to build frequent and infrequent decks; the paper relies on frequency differences to explain performance gaps.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Evaluating Creativity and Deception in Large Language Models: A Simulation Framework for Multi-Agent Balderdash." pith.science (2026). https://pith.science/paper/PAFWGQAP

@misc{pith2026241110422,
  author       = {Pith},
  title        = {Pith review of: Evaluating Creativity and Deception in Large Language Models: A Simulation Framework for Multi-Agent Balderdash},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PAFWGQAP}},
  note         = {Machine review of arXiv:2411.10422}
}

read the original abstract

Large Language Models (LLMs) have shown impressive capabilities in complex tasks and interactive environments, yet their creativity remains underexplored. This paper introduces a simulation framework utilizing the game Balderdash to evaluate both the creativity and logical reasoning of LLMs. In Balderdash, players generate fictitious definitions for obscure terms to deceive others while identifying correct definitions. Our framework enables multiple LLM agents to participate in this game, assessing their ability to produce plausible definitions and strategize based on game rules and history. We implemented a centralized game engine featuring various LLMs as participants and a judge LLM to evaluate semantic equivalence. Through a series of experiments, we analyzed the performance of different LLMs, examining metrics such as True Definition Ratio, Deception Ratio, and Correct Guess Ratio. The results provide insights into the creative and deceptive capabilities of LLMs, highlighting their strengths and areas for improvement. Specifically, the study reveals that infrequent vocabulary in LLMs' input leads to poor reasoning on game rules and historical context (https://github.com/ParsaHejabi/Simulation-Framework-for-Multi-Agent-Balderdash).

Figures

Figures reproduced from arXiv: 2411.10422 by the authors.

**Figure 3.** The prompt for generating deceiving defini [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

**Figure 4.** The prompts for generating definitions to [PITH_FULL_IMAGE:figures/full_fig_p011_4.png] view at source ↗

Figures from the paper (6 more)

**Figure 2.** Figure 2: The prompt for generating a correct definition [PITH_FULL_IMAGE:figures/full_fig_p011_2.png]

**Figure 6.** Figure 6: The prompts for the semantical equivalence judge, used in the game engine. [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]

**Figure 7.** Figure 7: The generate definition prompt, in two versions for asking the player to generate a definition for a given [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]

**Figure 8.** Figure 8: The full and mini history prompts, used for providing the performance history of each player in the game [PITH_FULL_IMAGE:figures/full_fig_p013_8.png]

**Figure 9.** Figure 9: The game rules prompt, in two versions for providing instructions on playing the game and the game’s [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

**Figure 10.** Figure 10: The vote on definitions prompt, in two versions for asking the player to choose the reference dictionary [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

37 extracted references · 10 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Qin Cai, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, ..., and Xiren Zhou. 2024. https://arxiv.org/abs/2404.14219 Phi-3 techni...

arXiv 2024
[4]

AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

2024
[5]

Steven Bird and Edward Loper. 2004. https://aclanthology.org/P04-3031 NLTK : The natural language toolkit . In Proceedings of the ACL Interactive Poster and Demonstration Sessions , pages 214--217, Barcelona, Spain. Association for Computational Linguistics

2004
[6]

Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or llms as the judge? a study on judgement biases. arXiv preprint arXiv:2402.10669

arXiv 2024
[7]

Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2023. Self-collaboration code generation via chatgpt. arXiv preprint arXiv:2304.07590

arXiv 2023
[8]

cat-shaped mug

Vishnu Sashank Dorbala, James F. Mullen, and Dinesh Manocha. 2024. https://doi.org/10.1109/LRA.2023.3346800 Can an embodied agent find your “cat-shaped mug”? llm-based zero-shot object navigation . IEEE Robotics and Automation Letters, 9(5):4083--4090

arXiv 2024

Show all 37 references

[9]

Thomas Mesnard Gemma Team, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, and et al. 2024. https://doi.org/10.34740/KAGGLE/M/3301 Gemma

2024 doi
[10]

Carlos G \'o mez-Rodr \' guez and Paul Williams. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.966 A confederacy of models: a comprehensive evaluation of LLM s on creative writing . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14504--...

2023 doi
[11]

Sylvain Gugger, Lysandre Debut, Thomas Wolf, Philipp Schmid, Zachary Mueller, Sourab Mangrulkar, Marc Sun, and Benjamin Bossan. 2022. Accelerate: Training and inference at scale made simple, efficient and adaptable. https://github.com/huggingface/accelerate

2022
[12]

Joy Paul Guilford. 1967. The nature of human intelligence. McGraw-Hill

1967
[13]

Chawla, Olaf Wiest, and Xiangliang Zhang

Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. 2024. https://arxiv.org/abs/2402.01680 Large language model based multi-agents: A survey of progress and challenges . Preprint, arXiv:2402.01680

2024 arXiv
[14]

Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. 2023. Metagpt: Meta programming for multi-agent collaborative framework. arXiv preprint arXiv:2308.00352

2023 arXiv
[15]

James Jhirad, Edison Marrese-Taylor, and Yutaka Matsuo. 2023. Evaluating large language models’ understanding of financial terminology via definition modeling. In Proceedings of the 13th International Joint Conference on Natural Language Processing and the 3rd Conference of th...

2023
[16]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

2023 arXiv
[17]

Cheongwoong Kang and Jaesik Choi. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.518 Impact of co-occurrence on factual knowledge of large language models . In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7721--7735, Singapore. Associatio...

2023 doi
[18]

Hanmi Lee, Wenqing Zhou, HongHong Bai, Weiran Meng, Tianli Zeng, Kaiping Peng, Song Tong, and Takatsune Kumada. 2023. Natural language processing algorithms for divergent thinking assessment. In 2023 ieee 6th eurasian conference on educational innovation (ecei), pages 198--202. IEEE

2023
[19]

Jonathan Light, Min Cai, Sheng Shen, and Ziniu Hu. 2023. https://arxiv.org/abs/2310.05036 Avalonbench: Evaluating llms playing the game of avalon . Preprint, arXiv:2310.05036

2023 arXiv
[20]

Merriam-Webster . 2024. Merriam-webster's dictionary api. https://dictionaryapi.com/. Accessed: 2024-04-18

2024
[21]

Gabriel Mukobi, Hannah Erlebach, Niklas Lauffer, Lewis Hammond, Alan Chan, and Jesse Clifton. 2024. https://openreview.net/forum?id=AKJLnDgzkm Welfare diplomacy: Benchmarking language model cooperation

2024
[22]

OpenAI . 2024. New embedding models and api updates. https://openai.com/index/new-embedding-models-and-api-updates/. Accessed: 2024-05-16

2024
[23]

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff ...

2024 arXiv
[24]

William Orwig, Emma R Edenbaum, Joshua D Greene, and Daniel L Schacter. 2024. The language of creativity: Evidence from humans and large language models. The Journal of Creative Behavior

2024
[25]

Oxford University Press . 2024. https://www.oxfordlearnersdictionaries.com/us/wordlist/american_english/oxford3000/ Oxford learner's dictionaries: Oxford 3000 word list . Accessed: 2024-04-18

2024
[26]

Chen Qian, Xin Cong, Cheng Yang, Weize Chen, Yusheng Su, Juyuan Xu, Zhiyuan Liu, and Maosong Sun. 2023. Communicative agents for software development. arXiv preprint arXiv:2307.07924

2023 arXiv
[27]

Ishika Singh, David Traum, and Jesse Thomason. 2024. Twostep: Multi-agent task planning using classical planners and large language models. arXiv preprint arXiv:2403.17246

2024 arXiv
[28]

Yurun Song, Junchen Zhao, and Lucia Specia. 2021. https://doi.org/10.18653/v1/2021.naacl-main.252 S ent S im: Crosslingual semantic evaluation of machine translation . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Ling...

2021 doi
[29]

Martin Trenkmann. 2023. https://ngrams.dev NGRAMS -- search the world's largest ngram dataset . Accessed on May 18, 2024

2023
[30]

Shenzhi Wang, Chang Liu, Zilong Zheng, Siyuan Qi, Shuo Chen, Qisen Yang, Andrew Zhao, Chaofei Wang, Shiji Song, and Gao Huang. 2023. https://arxiv.org/abs/2310.01320 Avalon's game of thoughts: Battle against deception through recursive contemplation . Preprint, arXiv:2310.01320

2023 arXiv
[31]

Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf Chain-of-thought prompting elicits reasoning...

2022
[32]

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

2020 doi
[33]

Yuzhuang Xu, Shuo Wang, Peng Li, Fuwen Luo, Xiaolong Wang, Weidong Liu, and Yang Liu. 2023. https://arxiv.org/abs/2309.04658 Exploring large language models for communication games: An empirical study on werewolf . Preprint, arXiv:2309.04658

2023 arXiv
[34]

Zelai Xu, Chao Yu, Fei Fang, Yu Wang, and Yi Wu. 2024. https://arxiv.org/abs/2310.18940 Language agents with reinforcement learning for strategic play in the werewolf game . Preprint, arXiv:2310.18940

2024 arXiv
[35]

Yunpu Zhao, Rui Zhang, Wenyi Li, Di Huang, Jiaming Guo, Shaohui Peng, Yifan Hao, Yuanbo Wen, Xing Hu, Zidong Du, et al. 2024. Assessing and understanding creativity in large language models. arXiv preprint arXiv:2401.12491

2024 arXiv
[36]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36

2024
[37]

Zhiling Zheng, Oufan Zhang, Ha L Nguyen, Nakul Rampal, Ali H Alawadhi, Zichao Rong, Teresa Head-Gordon, Christian Borgs, Jennifer T Chayes, and Omar M Yaghi. 2023. Chatgpt research group for optimizing the crystallinity of mofs and cofs. ACS Central Science, 9(11):2161--2170

2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorck, Sébastien Bubeck, Qin Cai, Martin Cai, Caio César Teodoro Mendes, Weizhu Chen, ..., and Xiren Zhou. 2024. https://arxiv.org/abs/2404.14219 Phi-3 techni...

arXiv 2024

[4] [4]

AI@Meta. 2024. https://github.com/meta-llama/llama3/blob/main/MODEL_CARD.md Llama 3 model card

2024

[5] [5]

Steven Bird and Edward Loper. 2004. https://aclanthology.org/P04-3031 NLTK : The natural language toolkit . In Proceedings of the ACL Interactive Poster and Demonstration Sessions , pages 214--217, Barcelona, Spain. Association for Computational Linguistics

2004

[6] [6]

Guiming Hardy Chen, Shunian Chen, Ziche Liu, Feng Jiang, and Benyou Wang. 2024. Humans or llms as the judge? a study on judgement biases. arXiv preprint arXiv:2402.10669

arXiv 2024

[7] [7]

Yihong Dong, Xue Jiang, Zhi Jin, and Ge Li. 2023. Self-collaboration code generation via chatgpt. arXiv preprint arXiv:2304.07590

arXiv 2023

[8] [8]

cat-shaped mug

Vishnu Sashank Dorbala, James F. Mullen, and Dinesh Manocha. 2024. https://doi.org/10.1109/LRA.2023.3346800 Can an embodied agent find your “cat-shaped mug”? llm-based zero-shot object navigation . IEEE Robotics and Automation Letters, 9(5):4083--4090

arXiv 2024