REVIEW 3 major objections 6 minor 43 references
The "LLM World of Words" English free association norms generated by large language models
T0 review · 3 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read LLM-generated free association norms are valid tools for studying semantic memory, because spreading-activation simulations on their networks replicate human priming effects and show similar gender stereotypes.
desk verdict Useful new resource, honest reporting, but the comparability claim is under-supported until prompt artifacts and generation details are addressed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the free-association network model of semantic memory: cue-response pairs collected in the norming task become a directed weighted graph (later made undirected) in which node activation spreads along weighted edges, decaying with distance, after a prime word is activated. Spreading activation is run for a number of time steps equal to twice the network diameter, and the final activation matrix is normalized first by column and then by row to factor out node centrality. This lets the authors measure the relational strength between any prime-target pair and compare it across human and LLM networks, producing the validation against behavioral reaction times and the bias quantification.
What would settle it
Remove every occurrence of the responses 'input', 'output', and 'association' from the Mistral data (the task-artifact words the paper documents) and rerun the spreading-activation validation: if the Spearman correlation with human reaction times drops below -0.5 or becomes non-significant, the apparent semantic signal in that network is largely carried by artifacts. Alternatively, rerun the entire pipeline with a different prompt phrase (e.g., 'write the first three words you think of') and compare the priming correlations; if they vary strongly with prompt wording, the norms are prompt-specific rather than a stable property of the models.
Extended reading notes
Core claim
The central claim is that free association norms elicited from LLMs by a simple prompt are not noise but a faithful signal of the models' implicit semantic associations, comparable in kind to human norms. Using the same cues as the human SWOW norms, the authors prompted three LLMs 100 times per cue and preprocessed the responses with lemmatization, spelling correction, and removal of cue-identical and duplicate responses. From these data they constructed undirected weighted networks in which nodes are words and edge weights are association frequencies, filtered to a common set of 11,545 cues and a largest connected component. Running spreading activation from prime nodes and measuring final activation of target nodes, they reproduced the human semantic priming effect in every LLM network, with effect sizes (0.86-0.87) close to the human effect (0.87) and Spearman correlations with lexical-decision reaction times all below -0.61 (p < 0.001). Applying the same machinery to gender-related primes and targets, they found stereotype-consistent pairs produce higher activation in all four networks, with the extent of bias differing by model. The paper concludes that the LWOW datasets are valid tools for investigating the structure of LLM semantic memory and for comparing implicit biases across humans and machines.
Load-bearing premise
The key assumption is that repeatedly asking an LLM to respond with one word and then cleaning the text yields a window into the model's underlying semantic associations, rather than a reflection of the prompt's wording, model-specific instruction-following quirks, or random generation.
Editorial extensions
If this is right
- The LWOW dataset makes it possible to ask, for the first time at this scale, whether LLM associative structures mirror human norms cue-by-cue: pairwise node and edge comparisons in the paper show overlapping lexicons but substantial structural divergence, with Llama3 closest in node overlap and Mistral sharing the most edges.
- Spreading-activation validation can serve as a model-agnostic benchmark for whether a given LLM's free associations encode human-like semantic relatedness, complementing embedding-based similarity measures.
- The gender-stereotype analysis provides a template for measuring implicit bias in LLMs from generated associations rather than from internal embeddings, allowing direct comparison to human implicit attitudes.
- Because each cue was probed 100 times, the dataset supports reliability analyses and can be used to study within-model consistency of associations, not just aggregate norms.
Reading between the lines
- A testable implication of the paper's framing is that LLM free associations reflect the same spreading-activation process proposed for human memory; an alternative interpretation is that the network similarity arises from shared statistical regularities in training text, not neural mechanisms. One way to separate these is to block or reweight high-frequency task artifacts (e.g., 'input', 'output',
- The validation uses human reaction times from a single lexical-decision dataset; extending it to multilingual or developmental norms would show whether the match is an artifact of the specific 50 prime-target pairs or generalizes.
- The reported Llama3 network has the largest node and edge counts among the LLMs, closest in size to the human network; this suggests model scale or training distribution may matter more than instruction-following ability for producing human-like associations, a hypothesis the paper does not test directly.
- A stronger validation would be a direct head-to-head prediction of human free association responses from LLM norms, e.g., ranking the most likely human responses to each cue; the paper does not report such a predictive overlap measure.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces LWOW, a dataset of over 3 million free-association responses generated by three LLMs (Mistral, Llama3, and Haiku) for the same cue words used in the human SWOW norms. For each of 12,282 cues, the authors prompted the LLM 100 times to obtain three responses per repetition, then preprocessed the data (lowercasing, lemmatization, blank-filling, response deduplication) to produce 11,545 cues per dataset. From these data they build directed-then-undirected weighted semantic networks for humans and each LLM, filter them (WordNet nodes, edge weight > 1, largest connected component), and validate the networks by simulating spreading activation on 50 prime-target pairs from the Semantic Priming Project, finding Spearman correlations around -0.6 between activation and human reaction times in all four networks. They also demonstrate a gender-stereotype analysis using 5 female/male primes and 25 gendered targets, reporting stereotype-consistent activation differences in all networks. The central claim is that LWOW is a valid, comparable free-association resource for studying semantic memory in LLMs.
Significance. If the validity claim holds, LWOW would be a valuable open resource for machine psychology, enabling large-scale comparisons of human and LLM semantic-network structure and implicit biases. The paper has clear strengths: it uses an independent external benchmark (Semantic Priming Project reaction times) and an established, independently developed tool (spreadr) for validation; the results are consistent across all four networks; the code and data are publicly available; and the gender-bias analysis provides a concrete, reproducible demonstration of the intended application. However, the validity of the dataset hinges on the untested assumption that repeated LLM prompting samples associative content rather than meta-task behavior. The paper itself documents that Mistral produced the high-frequency responses 'input', 'output', and 'association', which survive preprocessing and can survive network filtering; the validation sample of 50 prime-target pairs is too small and unrepresentative to rule out substantial contamination.
major comments (3)
- [Methods, Data generation] The exact prompt is not included in the manuscript; the sentence 'The following prompt was given to the LLMs:' is followed by a blank. In addition, the paper reports no model versions, sampling temperature, top-p, maximum token count, or API access dates for Mistral, Llama3, and Haiku. These details are essential for reproducibility of a dataset that is the paper's main contribution. Please include the full prompt and all generation parameters in the paper or in a stable appendix, not only in the GitHub repository.
- [Data processing] The authors state that responses like 'input', 'output', and 'association' appeared much more frequently in Mistral's data than in human data and that such responses are difficult to identify, so they were not removed from the preprocessed data. However, §Network construction removes only non-WordNet nodes and weight-1 edges; these meta-task words are in WordNet and can have edge weight > 1, so they survive the filtering. Table 1 shows that Mistral and Haiku have only 41,369 and 15,275 unique responses versus 116,640 for humans, indicating highly repetitive, potentially artifact-dominated output. The paper provides no quantitative audit of how many of the 3M+ responses are instruction-following artifacts, how their frequency varies by cue, or how many edges in the reduced networks are artifacts. Since the central claim is comparability to human free association norms, this omission is load-bearing: if many repetitions reflect task compliance rather than semantic content, the networks may measure instruction-following behavior, not LLM semantic memory. Please add an artifact-frequency analysis (e.g., per-cue counts of meta-task tokens) and a sensitivity check that re-runs the validation after removing artifact-dominated responses.
- [Technical Validation] The validation rests on 50 prime-target pairs from the Semantic Priming Project (Appendix Table 7). The authors do not justify why this subset was chosen, and 50 pairs are too few to establish that the networks are free of contamination, especially because these pairs are common semantic associates and are unlikely to be adjacent to artifact hubs such as 'input' or 'output'. The Spearman correlations of approximately -0.6 are encouraging but compatible with substantial artifact-driven edges elsewhere in the networks. Please validate on a larger and more diverse sample from the Semantic Priming Project (hundreds of pairs) and/or stratify the sample to include cue words whose responses are artifact-heavy, so that the validation can directly address the contamination concern raised in Major Comment 2.
minor comments (6)
- [Background & Summary] The phrase 'following the same methodology' overstates the comparability of the human and LLM tasks: humans provided responses in a continuous-association setting, while each LLM was prompted with a single instruction repeated 100 times. Please rephrase to 'modeled after' or 'following a similar protocol'.
- [Data processing] The step 'we ensured 100 repetitions per cue by adding blank responses when there were less than 100 repetitions' is not defined; please specify whether blanks are empty strings or a placeholder, and state whether they are excluded from the network construction (the WordNet filter presumably removes them, but this should be explicit).
- [Technical Validation] The text says 'we activated each prime from all 100 prime-target pairs' but the appendix lists 50 targets; please clarify that there are 100 primes (50 related and 50 unrelated) for 50 targets.
- [Table 5] The header 'Effect size (prime type)' is misleading; it should read 'Effect size (related vs. unrelated prime)' to reflect the comparison being made. The same issue appears in Table 6.
- [Usage Notes] Figure 5 is captioned 'Validation' but it displays the gender-bias results; please rename the caption to 'Gender bias' to avoid confusion with the Technical Validation section.
- [Appendix] Figures 6-8 (LLM heatmaps) are not discussed in the main text; please add a sentence in the Usage Notes describing the main similarities and differences between the human and LLM heatmaps.
Circularity Check
No significant circularity: LWOW validation rests on external benchmarks and an independent spreading-activation tool.
full rationale
The paper's central claim is that LLM-generated free association norms are comparable in structure and validity to human SWOW norms. The validation chain is genuinely external: (1) networks are built from the LWOW/SWOW norms by a standard cue-to-response edge construction; (2) semantic priming is simulated with the independently developed spreadr package (Siew 2019) using its default or documented parameter settings; (3) the simulated activation levels are compared against human reaction times from the separate Semantic Priming Project (Hutchison et al. 2013). No parameter is fitted to the benchmark and then reported as a prediction; the Spearman correlations and Wilcoxon effect sizes are out-of-sample outcomes of a fixed simulation pipeline. The gender-stereotype analysis is explicitly framed as a usage demonstration rather than a derivation of dataset validity. The only self-citations are to the authors' prior methodological work, e.g., setting activation iterations to two times network diameter (Samuel et al. 2023, with Stella as coauthor) and applying cognitive network science to LLMs (Abramski et al. 2023). These citations do not carry the load of the validation: they are ordinary methodological pointers, not uniqueness theorems or fitted inputs, and the same simulation and benchmarks would be available to any researcher. The paper's own admission that Mistral produced task-artifact responses such as 'input', 'output', and 'association' is a genuine data-quality limitation affecting construct validity, not a circularity: the 50-pair LDT validation may be too small to detect artifact contamination, but that is a correctness or generalizability risk rather than a self-referential derivation. No circular step can be exhibited with the required specificity.
Assumptions & free parameters
free parameters (3)
- Initial activation level of prime node =
number of nodes in the network
- Number of iterations for spreading activation =
two times the network diameter
- spreadr default parameters (retention, etc.) =
library defaults
assumptions (4)
- domain assumption Free associations are a valid measure of implicit semantic memory
- domain assumption Spreading activation in a network approximates semantic priming
- ad hoc to paper The 50 prime-target pairs from the Semantic Priming Project are representative of priming phenomena
- ad hoc to paper WordNet filtering and removal of weight-1 edges preserve meaningful semantic structure
Cite this review
Pith. "Pith review of The "LLM World of Words" English free association norms generated by large language models." pith.science (2026). https://pith.science/paper/EKPKDOSF
@misc{pith2026241201330,
author = {Pith},
title = {Pith review of: The "LLM World of Words" English free association norms generated by large language models},
year = {2026},
howpublished = {\url{https://pith.science/paper/EKPKDOSF}},
note = {Machine review of arXiv:2412.01330}
}
read the original abstract
Free associations have been extensively used in cognitive psychology and linguistics for studying how conceptual knowledge is organized. Recently, the potential of applying a similar approach for investigating the knowledge encoded in LLMs has emerged, specifically as a method for investigating LLM biases. However, the absence of large-scale LLM-generated free association norms that are comparable with human-generated norms is an obstacle to this new research direction. To address this limitation, we create a new dataset of LLM-generated free association norms modeled after the "Small World of Words" (SWOW) human-generated norms consisting of approximately 12,000 cue words. We prompt three LLMs, namely Mistral, Llama3, and Haiku, with the same cues as those in the SWOW norms to generate three novel comparable datasets, the "LLM World of Words" (LWOW). Using both SWOW and LWOW norms, we construct cognitive network models of semantic memory that represent the conceptual knowledge possessed by humans and LLMs. We demonstrate how these datasets can be used for investigating implicit biases in humans and LLMs, such as the harmful gender stereotypes that are prevalent both in society and LLM outputs.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
De Deyne, S., Navarro, D. J. & Storms, G. Better explanations of lexical and semantic cognition using networks derived from continued rather than single-word associations. Behav. research methods 45, 480–498 (2013)
work page 2013
-
[2]
Kenett, Y . N., Levi, E., Anaki, D. & Faust, M. The semantic distance task: Quantifying semantic distance with semantic network path length. J. Exp. Psychol. Learn. Mem. Cogn. 43, 1470 (2017). 14/16
work page 2017
-
[3]
Words in the mind: An introduction to the mental lexicon(John Wiley & Sons, 2012)
Aitchison, J. Words in the mind: An introduction to the mental lexicon(John Wiley & Sons, 2012)
work page 2012
-
[4]
Vankrunkelsven, H., Verheyen, S., Storms, G. & De Deyne, S. Predicting lexical norms: A comparison between a word association model and text-based word co-occurrence models. J. cognition 1 (2018)
work page 2018
-
[5]
Samuel, G., Stella, M., Beaty, R. E. & Kenett, Y . N. Predicting openness to experience via a multiplex cognitive network approach. J. Res. Pers. 104, 104369 (2023)
work page 2023
-
[6]
Shiffrin, R. & Mitchell, M. Probing the psychology of ai models. Proc. Natl. Acad. Sci. 120, e2300963120 (2023)
work page 2023
-
[7]
Binz, M. & Schulz, E. Using cognitive psychology to understand gpt-3. Proc. Natl. Acad. Sci. 120, e2218523120 (2023)
work page 2023
-
[8]
Steyvers, M. & Tenenbaum, J. B. The large-scale structure of semantic networks: Statistical analyses and a model of semantic growth. Cogn. science 29, 41–78 (2005)
work page 2005
Show all 43 references
-
[9]
& Siew, C
Stella, M., De Nigris, S., Aloric, A. & Siew, C. S. Forma mentis networks quantify crucial differences in stem perception between students and experts. PloS one 14, e0222870 (2019)
2019
-
[10]
& Storms, G
De Deyne, S. & Storms, G. Word associations: Network and semantic properties. Behav. research methods 40, 213–231 (2008)
2008
-
[11]
small world of words
De Deyne, S., Navarro, D. J., Perfors, A., Brysbaert, M. & Storms, G. The “small world of words” english word association norms for over 12,000 cue words. Behav. research methods 51, 987–1006 (2019)
2019
-
[12]
& Bentz, C
Hill, F., Korhonen, A. & Bentz, C. A quantitative empirical analysis of the abstract/concrete distinction. Cogn. science 38, 162–177 (2014)
2014
-
[13]
& Asendorpf, J
Schnabel, K. & Asendorpf, J. B. Free associations as a measure of stable implicit attitudes. Eur. J. Pers. 27, 39–50 (2013)
2013
-
[14]
S., Stella, M
Citraro, S., Vitevitch, M. S., Stella, M. & Rossetti, G. Feature-rich multiplex lexical networks reveal mental strategies of early language learning. Sci. Reports 13, 1474 (2023)
2023
-
[15]
Kenett, Y . N. & Austerweil, J. L. Examining search processes in low and high creative individuals with random walks. In CogSci, vol. 8, 313–318 (2016)
2016
-
[16]
Beaty, R. E. & Kenett, Y . N. Associative thinking at the core of creativity.Trends Cogn. Sci. (2023)
2023
-
[17]
Benedek, M. et al. How semantic memory structure and intelligence contribute to creative thought: A network science approach. Think. & Reason. 23, 158–183 (2017)
2017
-
[18]
N., Gold, R
Kenett, Y . N., Gold, R. & Faust, M. The hyper-modular associative mind: a computational analysis of associative responses of persons with asperger syndrome. Lang. Speech 59, 297–317 (2016)
2016
-
[19]
From word types to tokens and back: A survey of approaches to word meaning representation and interpretation
Apidianaki, M. From word types to tokens and back: A survey of approaches to word meaning representation and interpretation. Comput. Linguist. 1–60 (2022)
2022
-
[20]
Rodriguez, M. A. & Merlo, P. Word associations and the distance properties of context-aware word embeddings. In Proceedings of the 24th Conference on Computational Natural Language Learning, 376–385 (2020)
2020
-
[21]
& Barbosa, D
Yao, P., Renwick, T. & Barbosa, D. Wordties: Measuring word associations in language models via constrained sampling. In Findings of the Association for Computational Linguistics: EMNLP 2022, 5959–5970 (2022)
2022
-
[22]
Y ., Saligrama, V
Bolukbasi, T., Chang, K.-W., Zou, J. Y ., Saligrama, V . & Kalai, A. T. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. Adv. neural information processing systems 29 (2016)
2016
-
[23]
Kurita, K., Vyas, N., Pareek, A., Black, A. W. & Tsvetkov, Y . Measuring bias in contextualized word representations. arXiv preprint arXiv:1906.07337 (2019)
2019 arXiv
-
[24]
C., Tsvetkov, Y
Manzini, T., Lim, Y . C., Tsvetkov, Y . & Black, A. W. Black is to criminal as caucasian is to police: Detecting and removing multiclass bias in word embeddings. arXiv preprint arXiv:1904.04047 (2019)
2019 arXiv
-
[25]
Caliskan, A., Bryson, J. J. & Narayanan, A. Semantics derived automatically from language corpora contain human-like biases. Science 356, 183–186 (2017)
2017
-
[26]
& Cardie, C
Bommasani, R., Davis, K. & Cardie, C. Interpreting pretrained contextualized representations via reductions to static embeddings. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 4758–4781 (2020)
2020
-
[27]
G., McGhee, D
Greenwald, A. G., McGhee, D. E. & Schwartz, J. L. Measuring individual differences in implicit cognition: the implicit association test. J. personality social psychology 74, 1464 (1998)
1998
-
[28]
Srivastava, A. et al. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models. arXiv preprint arXiv:2206.04615 (2022). 15/16
2022 arXiv
-
[29]
Machine psychology: Investigating emergent capabilities and behavior in large language models using psychological methods
Hagendorff, T. Machine psychology: Investigating emergent capabilities and behavior in large language models using psychological methods. arXiv preprint arXiv:2303.13988 (2023)
2023 arXiv
-
[30]
The fluency-based semantic network of llms differs from humans
Wang, Y .et al. The fluency-based semantic network of llms differs from humans. Comput. Hum. Behav. Artif. Humans 100103 (2024)
2024
-
[31]
Suresh, S. et al. Conceptual structure coheres in human cognition but not in large language models. arXiv preprint arXiv:2304.02754 (2023)
2023 arXiv
-
[32]
& Kosinski, M
Digutsch, J. & Kosinski, M. Overlap in meaning is a stronger predictor of semantic activation in gpt-3 than in humans. Sci. Reports 13, 5035 (2023)
2023
-
[33]
& Stella, M
Abramski, K., Citraro, S., Lombardi, L., Rossetti, G. & Stella, M. Cognitive network science reveals bias in gpt-3, gpt-3.5 turbo, and gpt-4 mirroring math anxiety in high-school students. Big Data Cogn. Comput. 7, 124 (2023)
2023
-
[34]
L., McEvoy, C
Nelson, D. L., McEvoy, C. L. & Schreiber, T. A. The university of south florida free association, rhyme, and word fragment norms. Behav. Res. Methods, Instruments, & Comput. 36, 402–407 (2004)
2004
-
[35]
Wilson, M., Kiss, G. et al. Eat: The edinburgh associative corpus. Oxf. Text Arch. Core Collect.(1988)
1988
-
[36]
Miller, G. A. Wordnet: a lexical database for english. Commun. ACM 38, 39–41 (1995)
1995
-
[37]
Siew, C. S. spreadr: An r package to simulate spreading activation in a network. Behav. Res. Methods 51, 910–929 (2019)
2019
-
[38]
Hutchison, K. A. et al. The semantic priming project. Behav. research methods 45, 1099–1114 (2013)
2013
-
[39]
Collins, A. M. & Loftus, E. F. A spreading-activation theory of semantic processing. Psychol. review 82, 407 (1975)
1975
-
[40]
Neely, J. H. Semantic priming effects in visual word recognition: A selective review of current findings and theories. Basic processes reading 264–336 (2012)
2012
-
[41]
McNamara, T. P. Semantic priming: Perspectives from memory and word recognition(Psychology Press, 2005)
2005
-
[42]
Collins, A. M. & Quillian, M. R. Retrieval time from semantic memory. J. verbal learning verbal behavior 8, 240–247 (1969)
1969
-
[43]
& Kay, A
Gaucher, D., Friesen, J. & Kay, A. C. Evidence that gendered wording in job advertisements exists and sustains gender inequality. J. personality social psychology 101, 109 (2011). Author contributions statement K.A., G.R. and M.S. conceived the experiments, K.A., R.I. and G.R....
2011
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.