{"id":"b89ef53f-336c-47aa-9e18-07a783be1155","arxiv_id":"2412.01330","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new dataset of 3+ million free association responses from three LLMs, matched to human norms, with validation showing human-like semantic priming and gender bias patterns.","lead":"Researchers asked three large language models to produce free word associations for 12,000 cue words, creating a new dataset called LWOW. The dataset lets scientists compare how humans and AI models organize concepts, including shared gender stereotypes.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Prompt-artifact responses may contaminate LLM association networks; the 50-pair LDT validation cannot detect this because it samples too few cue-target pairs.","rationale":"The reader's weakest assumption—that repeated prompting yields a faithful sample of implicit semantic associations after preprocessing—is exactly the load-bearing condition. The authors' own admission about Mistral artifacts, combined with the low unique-response counts for Mistral and Haiku, makes this concern concrete. The network filters (WordNet membership, weight>1) do not remove high-frequency artifact words because words like 'input' are valid English words and can exceed weight 1 when repeatedly generated. The technical validation uses only 50 prime-target pairs from a single dataset; these pairs are semantically related/unrelated words (e.g., 'doctor-nurse') that are likely not connected to artifact hubs. Success on these pairs therefore does not demonstrate that the full 12,000-cue, 3-million-response dataset reflects semantic association rather than prompt-instruction-following. The proposed test directly quantifies artifact prevalence and checks whether removing artifacts changes the validation outcome. If the correlations survive, the concern is empirically refuted; if not, the conditional verdict should be strengthened toward revision. Because the data and code are public, this is testable without new data collection. I agree with the reader's conditional verdict and see no reason to change it, but the artifact analysis is the specific gap that must be closed.","tokens_in":12483,"tokens_out":3557,"duration_ms":32529,"concrete_test":"Using the public LWOW CSVs, (1) count occurrences of meta-task responses ('input', 'output', 'association', 'word', 'response', 'prompt', plus morphological variants) for each model and for the Human dataset, reporting total frequency and number of distinct cue words for which each appears. (2) Rebuild the reduced networks after removing all such meta-task responses before the WordNet filter, then rerun the spreadr LDT analysis with the same 50 prime-target pairs. If the Spearman correlations for Mistral and Haiku remain within about 0.1 of their original values and stay significant at p<0.001, artifacts are not driving the validation; if they drop or lose significance, the dataset's central validity claim requires revision. Also report the number of nodes and edges removed by this filter.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that LWOW is a valid, comparable free-association resource for studying semantic memory in LLMs. This requires LLM repetitions to sample semantic associations rather than meta-task behaviors. The authors themselves report that Mistral produced 'input', 'output', and 'association' far more often than humans (Data processing section), and such tokens survive preprocessing because they are WordNet words and can have edge weight greater than 1 in network construction. Table 1 shows Mistral and Haiku have only 41,369 and 15,275 unique responses versus 116,640 for humans, indicating repetitive, potentially artifact-dominated output. The validation (Technical Validation section) relies on only 50 prime-target pairs from one lexical decision dataset (Appendix Table 7). These 50 pairs are unlikely to be adjacent to artifact hubs, so Spearman correlations near -0.6 can coexist with networks substantially contaminated by instruction-following responses. The paper provides no quantitative assessment of how many of the 3M+ responses are meta-task artifacts, how their frequency varies by cue, or how much they shape the reduced networks. Thus the comparability claim is not secured for the full dataset.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LWOW, a dataset of over 3 million free-association responses generated by three LLMs (Mistral, Llama3, and Haiku) for the same cue words used in the human SWOW norms. For each of 12,282 cues, the authors prompted the LLM 100 times to obtain three responses per repetition, then preprocessed the data (lowercasing, lemmatization, blank-filling, response deduplication) to produce 11,545 cues per dataset. From these data they build directed-then-undirected weighted semantic networks for humans and each LLM, filter them (WordNet nodes, edge weight > 1, largest connected component), and validate the networks by simulating spreading activation on 50 prime-target pairs from the Semantic Priming Project, finding Spearman correlations around -0.6 between activation and human reaction times in all four networks. They also demonstrate a gender-stereotype analysis using 5 female/male primes and 25 gendered targets, reporting stereotype-consistent activation differences in all networks. The central claim is that LWOW is a valid, comparable free-association resource for studying semantic memory in LLMs.","tokens_in":12761,"tokens_out":6577,"duration_ms":57261,"significance":"If the validity claim holds, LWOW would be a valuable open resource for machine psychology, enabling large-scale comparisons of human and LLM semantic-network structure and implicit biases. The paper has clear strengths: it uses an independent external benchmark (Semantic Priming Project reaction times) and an established, independently developed tool (spreadr) for validation; the results are consistent across all four networks; the code and data are publicly available; and the gender-bias analysis provides a concrete, reproducible demonstration of the intended application. However, the validity of the dataset hinges on the untested assumption that repeated LLM prompting samples associative content rather than meta-task behavior. The paper itself documents that Mistral produced the high-frequency responses 'input', 'output', and 'association', which survive preprocessing and can survive network filtering; the validation sample of 50 prime-target pairs is too small and unrepresentative to rule out substantial contamination.","major_comments":[{"comment":"The exact prompt is not included in the manuscript; the sentence 'The following prompt was given to the LLMs:' is followed by a blank. In addition, the paper reports no model versions, sampling temperature, top-p, maximum token count, or API access dates for Mistral, Llama3, and Haiku. These details are essential for reproducibility of a dataset that is the paper's main contribution. Please include the full prompt and all generation parameters in the paper or in a stable appendix, not only in the GitHub repository.","section":"Methods, Data generation"},{"comment":"The authors state that responses like 'input', 'output', and 'association' appeared much more frequently in Mistral's data than in human data and that such responses are difficult to identify, so they were not removed from the preprocessed data. However, §Network construction removes only non-WordNet nodes and weight-1 edges; these meta-task words are in WordNet and can have edge weight > 1, so they survive the filtering. Table 1 shows that Mistral and Haiku have only 41,369 and 15,275 unique responses versus 116,640 for humans, indicating highly repetitive, potentially artifact-dominated output. The paper provides no quantitative audit of how many of the 3M+ responses are instruction-following artifacts, how their frequency varies by cue, or how many edges in the reduced networks are artifacts. Since the central claim is comparability to human free association norms, this omission is load-bearing: if many repetitions reflect task compliance rather than semantic content, the networks may measure instruction-following behavior, not LLM semantic memory. Please add an artifact-frequency analysis (e.g., per-cue counts of meta-task tokens) and a sensitivity check that re-runs the validation after removing artifact-dominated responses.","section":"Data processing"},{"comment":"The validation rests on 50 prime-target pairs from the Semantic Priming Project (Appendix Table 7). The authors do not justify why this subset was chosen, and 50 pairs are too few to establish that the networks are free of contamination, especially because these pairs are common semantic associates and are unlikely to be adjacent to artifact hubs such as 'input' or 'output'. The Spearman correlations of approximately -0.6 are encouraging but compatible with substantial artifact-driven edges elsewhere in the networks. Please validate on a larger and more diverse sample from the Semantic Priming Project (hundreds of pairs) and/or stratify the sample to include cue words whose responses are artifact-heavy, so that the validation can directly address the contamination concern raised in Major Comment 2.","section":"Technical Validation"}],"minor_comments":[{"comment":"The phrase 'following the same methodology' overstates the comparability of the human and LLM tasks: humans provided responses in a continuous-association setting, while each LLM was prompted with a single instruction repeated 100 times. Please rephrase to 'modeled after' or 'following a similar protocol'.","section":"Background & Summary"},{"comment":"The step 'we ensured 100 repetitions per cue by adding blank responses when there were less than 100 repetitions' is not defined; please specify whether blanks are empty strings or a placeholder, and state whether they are excluded from the network construction (the WordNet filter presumably removes them, but this should be explicit).","section":"Data processing"},{"comment":"The text says 'we activated each prime from all 100 prime-target pairs' but the appendix lists 50 targets; please clarify that there are 100 primes (50 related and 50 unrelated) for 50 targets.","section":"Technical Validation"},{"comment":"The header 'Effect size (prime type)' is misleading; it should read 'Effect size (related vs. unrelated prime)' to reflect the comparison being made. The same issue appears in Table 6.","section":"Table 5"},{"comment":"Figure 5 is captioned 'Validation' but it displays the gender-bias results; please rename the caption to 'Gender bias' to avoid confusion with the Technical Validation section.","section":"Usage Notes"},{"comment":"Figures 6-8 (LLM heatmaps) are not discussed in the main text; please add a sentence in the Usage Notes describing the main similarities and differences between the human and LLM heatmaps.","section":"Appendix"}],"recommendation":"major_revision","confidential_remarks":"The dataset has clear potential value for the machine-psychology community, and the authors have made the code and data publicly available. However, for a data descriptor, the missing prompt and generation parameters are a serious reproducibility gap, and the artifact-contamination concern, which the authors themselves acknowledge in the Data processing section, is not addressed quantitatively. The current 50-pair validation is too weak to support the central comparability claim. I would encourage the editor to request a revision that includes a full artifact audit and a more extensive validation, as these are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a genuinely useful public resource: the first large-scale LLM free association norms aligned with the SWOW cues, with code and data on GitHub. Second, the central claim that these are directly comparable to human norms is only partially secured. The authors are honest about the prompt artifacts, but they don't quantify how much of the output is task-related chaff, and the 50-pair validation isn't enough to rule out contamination.\n\nWhat's new: 12k+ cues, 100 repetitions x 3 responses across three LLMs, matching SWOW structure. The dataset is the first openly available thing of its kind. The external validation is a real strength: spreading activation on all four networks correlates with human reaction times from the Semantic Priming Project (Spearman around -0.6, p<0.001), with effect sizes comparable to the human benchmark. Using an external benchmark rather than self-similarity is the right move. The bias demonstration is clearly framed as a usage note, not a headline result.\n\nSoft spots, in order. Missing generation details: no temperature, no model versions, no API dates. That matters for reproducibility of a dataset paper. More importantly, the actual prompt is not shown in the manuscript—only a placeholder. The entire dataset depends on that prompt, and it's absent. The stress-test concern about artifacts is valid: Mistral and Haiku produce far fewer unique responses (41k and 15k vs 116k for humans), and 'input', 'output', 'association' are acknowledged as high-frequency task artifacts. Because those words are WordNet words and have edge weight >1, they survive the network filter. The 50 validation pairs are real words like 'dill', 'coral', 'toast'; they are unlikely to sit next to artifact hubs, so the correlation can hold even if the network is locally dominated by instruction-following responses. The paper would be much stronger with a quantitative artifact audit—how many responses are meta-task words, how they distribute by cue, and whether removing them changes the validation or the bias results.\n\nVerdict: worth reviewing seriously. Given the missing prompt and generation parameters, plus the artifact question, I'd ask for major revision. The dataset is valuable enough that this should go forward, but the comparability claim needs more support.","headline":"Useful new resource, honest reporting, but the comparability claim is under-supported until prompt artifacts and generation details are addressed.","tokens_in":13228,"tokens_out":2785,"would_cite":true,"duration_ms":24210,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-generated free association norms are valid tools for studying semantic memory, because spreading-activation simulations on their networks replicate human priming effects and show similar gender stereotypes.","keywords":["free association norms","semantic memory","large language models","cognitive network science","spreading activation","semantic priming","gender stereotypes","machine psychology"],"falsifier":"Remove every occurrence of the responses 'input', 'output', and 'association' from the Mistral data (the task-artifact words the paper documents) and rerun the spreading-activation validation: if the Spearman correlation with human reaction times drops below -0.5 or becomes non-significant, the apparent semantic signal in that network is largely carried by artifacts. Alternatively, rerun the entire pipeline with a different prompt phrase (e.g., 'write the first three words you think of') and compare the priming correlations; if they vary strongly with prompt wording, the norms are prompt-specific rather than a stable property of the models.","tokens_in":12321,"feed_emoji":"🧠","tokens_out":6529,"duration_ms":53930,"temperature":0.7,"pith_summary":"The paper presents LWOW, a dataset of free association norms generated by three large language models (Mistral, Llama3, Haiku) using the same 12,000+ cue words and the same repeated-response design as the human 'Small World of Words' norms. It aims to establish that these LLM-generated norms are a valid, directly comparable resource for studying how conceptual knowledge is organized in language models. To support this, the authors build weighted network models of semantic memory from the human and LLM norms and run spreading-activation simulations that replicate semantic priming: targets receive higher activation after related primes than unrelated primes, and activation levels correlate with human reaction times (around -0.6, all p < 0.001). They also show that the gender-stereotype patterns measured in the LLM networks resemble those in the human network, suggesting the dataset can be used to probe implicit bias in models. The paper's contribution is a new, publicly available dataset plus a validation pipeline that adapts classic cognitive-network methods to LLMs.","feed_headline":"LLM word associations pass the semantic priming test","feed_subtitle":"Over 3 million LLM responses reproduce human semantic priming effects and reveal comparable gender stereotypes.","key_machinery":"The central machinery is the free-association network model of semantic memory: cue-response pairs collected in the norming task become a directed weighted graph (later made undirected) in which node activation spreads along weighted edges, decaying with distance, after a prime word is activated. Spreading activation is run for a number of time steps equal to twice the network diameter, and the final activation matrix is normalized first by column and then by row to factor out node centrality. This lets the authors measure the relational strength between any prime-target pair and compare it across human and LLM networks, producing the validation against behavioral reaction times and the bias quantification.","core_discovery":"The central claim is that free association norms elicited from LLMs by a simple prompt are not noise but a faithful signal of the models' implicit semantic associations, comparable in kind to human norms. Using the same cues as the human SWOW norms, the authors prompted three LLMs 100 times per cue and preprocessed the responses with lemmatization, spelling correction, and removal of cue-identical and duplicate responses. From these data they constructed undirected weighted networks in which nodes are words and edge weights are association frequencies, filtered to a common set of 11,545 cues and a largest connected component. Running spreading activation from prime nodes and measuring final activation of target nodes, they reproduced the human semantic priming effect in every LLM network, with effect sizes (0.86-0.87) close to the human effect (0.87) and Spearman correlations with lexical-decision reaction times all below -0.61 (p < 0.001). Applying the same machinery to gender-related primes and targets, they found stereotype-consistent pairs produce higher activation in all four networks, with the extent of bias differing by model. The paper concludes that the LWOW datasets are valid tools for investigating the structure of LLM semantic memory and for comparing implicit biases across humans and machines.","pith_inferences":["A testable implication of the paper's framing is that LLM free associations reflect the same spreading-activation process proposed for human memory; an alternative interpretation is that the network similarity arises from shared statistical regularities in training text, not neural mechanisms. One way to separate these is to block or reweight high-frequency task artifacts (e.g., 'input', 'output',","The validation uses human reaction times from a single lexical-decision dataset; extending it to multilingual or developmental norms would show whether the match is an artifact of the specific 50 prime-target pairs or generalizes.","The reported Llama3 network has the largest node and edge counts among the LLMs, closest in size to the human network; this suggests model scale or training distribution may matter more than instruction-following ability for producing human-like associations, a hypothesis the paper does not test directly.","A stronger validation would be a direct head-to-head prediction of human free association responses from LLM norms, e.g., ranking the most likely human responses to each cue; the paper does not report such a predictive overlap measure."],"forward_implications":["The LWOW dataset makes it possible to ask, for the first time at this scale, whether LLM associative structures mirror human norms cue-by-cue: pairwise node and edge comparisons in the paper show overlapping lexicons but substantial structural divergence, with Llama3 closest in node overlap and Mistral sharing the most edges.","Spreading-activation validation can serve as a model-agnostic benchmark for whether a given LLM's free associations encode human-like semantic relatedness, complementing embedding-based similarity measures.","The gender-stereotype analysis provides a template for measuring implicit bias in LLMs from generated associations rather than from internal embeddings, allowing direct comparison to human implicit attitudes.","Because each cue was probed 100 times, the dataset supports reliability analyses and can be used to study within-model consistency of associations, not just aggregate norms."],"supporting_citations":[{"why":"Provides the SWOW human free association norms, the cue words, and the human responses that LWOW is modeled after.","marker":"[11]"},{"why":"Supplies the spreading-activation simulation algorithm (the spreadr R package) used to validate the networks.","marker":"[37]"},{"why":"Provides the lexical-decision-task reaction times for related and unrelated prime-target pairs used as the behavioral benchmark.","marker":"[38]"},{"why":"Establishes that free associations measure stable implicit attitudes, justifying the gender-bias analysis.","marker":"[13]"},{"why":"Supplies the list of female- and male-related target adjectives used to probe gender stereotypes.","marker":"[43]"},{"why":"WordNet is used for lemmatization and for filtering nonsensical responses during preprocessing.","marker":"[36]"},{"why":"Articulates the 'machine psychology' approach of using cognitive tasks to probe LLM behavior, motivating the prompting methodology.","marker":"[29]"}],"fun_headline_variants":["LLM free associations mirror human semantic priming","LLMs reveal human-like semantic priming and stereotypes","LLM word norms reproduce priming, expose biases"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The key assumption is that repeatedly asking an LLM to respond with one word and then cleaning the text yields a window into the model's underlying semantic associations, rather than a reflection of the prompt's wording, model-specific instruction-following quirks, or random generation.","fun_headline_variants_meta":{"raw":{"variants":["LLM free associations mirror human semantic priming","LLMs reveal human-like semantic priming and stereotypes","LLM word norms reproduce priming, expose biases"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000697,"raw_usage":{"total_tokens":3178,"prompt_tokens":1000,"completion_tokens":2178,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":616,"completion_tokens_details":{"reasoning_tokens":2132}},"tokens_in":616,"tokens_out":2178,"duration_ms":14686,"temperature":1.0,"reasoning_tokens":2132,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T04:27:31.290113+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Remove every occurrence of the responses 'input', 'output', and 'association' from the Mistral data (the task-artifact words the paper documents) and rerun the spreading-activation validation: if the Spearman correlation with human reaction times drops below -0.5 or becomes non-significant, the apparent semantic signal in that network is largely carried by artifacts. Alternatively, rerun the entire pipeline with a different prompt phrase (e.g., 'write the first three words you think of') and compare the priming correlations; if they vary strongly with prompt wording, the norms are prompt-specific rather than a stable property of the models.","supporting_citations":[{"cited_title":"small world of words","cited_arxiv_id":null,"evidence_quote":"Provides the SWOW human free association norms, the cue words, and the human responses that LWOW is modeled after."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the spreading-activation simulation algorithm (the spreadr R package) used to validate the networks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the lexical-decision-task reaction times for related and unrelated prime-target pairs used as the behavioral benchmark."},{"cited_title":"& Asendorpf, J","cited_arxiv_id":null,"evidence_quote":"Establishes that free associations measure stable implicit attitudes, justifying the gender-bias analysis."},{"cited_title":"& Kay, A","cited_arxiv_id":null,"evidence_quote":"Supplies the list of female- and male-related target adjectives used to probe gender stereotypes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"WordNet is used for lemmatization and for filtering nonsensical responses during preprocessing."}],"review_version":1}