{"id":"bc8046b7-6ffb-416d-a8c3-44d2577b051b","arxiv_id":"2509.14405","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A practical guide and software framework for generating psycholinguistic word norms with LLMs, validated against human ratings, with a case study on English familiarity.","lead":"This paper is a practical guide to using large language models to estimate psychological properties of words, like how familiar a word is, instead of collecting ratings from people. It includes a software framework and shows that fine-tuning a model on a few thousand human ratings gets close to human-level agreement.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Case-study correlations are computed against MRC/Glasgow norms that may be in GPT-4o-mini's training data, so the 0.9 fine-tuned result and the 'higher than Glasgow-MRC' claim are not yet evidence of generalization to new words.","rationale":"The reader's conditional verdict is appropriate, and the most load-bearing assumption is the one the reader identified: validation correlations on legacy human datasets do not establish generalization unless contamination can be ruled out. I do not see an internal contradiction in the guide; on the contrary, the authors repeatedly recommend collecting new human validation data, recommend fully-open models when possible, and provide a framework plus code. Those parts of the paper are valuable independently of the case-study numbers. The problem is that the case study, which makes the abstract's 0.8/0.9 claims concrete, does not follow its own strongest advice. Since GPT-4o-mini is proprietary, the contamination question cannot be settled by inspection, and the paper offers no new human data or membership-inference check. The comparison to the Glasgow-MRC correlation does not remove this threat, because a model fine-tuned on the mean of those two datasets could plausibly correlate with each at or above their inter-correlation even on held-out words; the more serious question is whether those held-out words were, in effect, already seen during pretraining. The lack of confidence intervals and the selection of the best fine-tuning target on the test set are secondary but real: they make the reported improvements less precise than the prose suggests. A single new-data validation would settle the main issue. If the new-data correlation remains around 0.9, the central claim is strongly supported; if it falls, the paper's practical guidance may still be sound, but the headline empirical claim should be downgraded or re-scoped to 'validated on legacy norms, pending new-data checks.' That is exactly the kind of condition the CONDITIONAL verdict should carry.","tokens_in":29253,"tokens_out":7797,"duration_ms":71468,"concrete_test":"Collect new human familiarity ratings for 500-1000 English words that are absent from MRC/Glasgow and postdate the model's training cutoff (or are otherwise verifiably not in GPT-4o-mini's training data), using the paper's own rating protocol (A.1.1). Apply the exact best prompt (e.g., few-shot v09 with logprobs) and the best mean-fine-tuned GPT-4o-mini model from A.4.3 to these words. If the Spearman correlation with the new human ratings is near 0.9 and above the human inter-rater baseline, the generalization claim survives; if it drops below ~0.8, the reported case-study numbers are substantially explained by contamination or overfitting. If new human data are not feasible, a membership-inference audit (Golchin & Surdeanu, 2023) on the MRC/Glasgow items is a weaker but informative substitute.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline numerical claims rest entirely on the English familiarity case study (Appendix A), where GPT-4o-mini is evaluated against the MRC (Coltheart et al., 1981) and Glasgow (Scott et al., 2019) norms. These datasets are old, widely used, and freely available online; GPT-4o-mini's training data are undisclosed. The paper itself flags this exact threat: 'good performance on that dataset cannot necessarily be generalized to other words' (Section 4), and data contamination is listed among the central pitfalls (Section 7.1) with membership inference as a suggested check. However, the case study performs no such audit and collects no new human ratings. Therefore the abstract's Spearman ~0.8/0.9 values, and especially the A.4.3 statement that the best fine-tuned model 'achieves a higher correlation than the one observed between Glasgow and MRC,' are equally compatible with the model having memorized the evaluation items. A secondary problem is that the best of three fine-tuning targets (Glasgow, MRC, mean) was selected after evaluating on the same 1045-word test set, and no confidence intervals are reported; this inflates the reported gain, though it is not as fundamental as the contamination threat.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper is a practical methodological guide to estimating word-level psycholinguistic norms with LLMs. It covers model selection, API versus web interfaces, prompt engineering, logprob-based aggregation, fine-tuning, validation against human norms, and a list of pitfalls and open questions. The authors also provide a Python framework and a case study on English familiarity in which base GPT-4o-mini estimates correlate at Spearman about 0.8 with MRC and Glasgow human norms, and fine-tuning increases this to about 0.9, with the best fine-tuned model reportedly exceeding the correlation between the Glasgow and MRC datasets themselves. The contribution is framed as a reference guide plus an illustrative workflow, not as a new full-scale norming database.","tokens_in":29624,"tokens_out":6824,"duration_ms":59288,"significance":"If the headline case-study numbers were trustworthy, this would be a genuinely useful contribution. The paper ships a reproducible framework with code and data, gives concrete and actionable advice on prompts, logprobs, fine-tuning, and validation, and is unusually candid about dead ends and limitations. The explicit emphasis on human gold-standard validation, inter-human baselines, and data contamination is a strength, and the case study is clearly described in enough detail to be replicated. The significance is contingent, however, on the validity of the 0.8/0.9 correlation claims, which currently rest on comparisons with old, widely available norms under an undisclosed training-data regime, with prompt and fine-tuning-target selection performed on the same evaluation sets.","major_comments":[{"comment":"The headline correlations (0.8 base, 0.9 fine-tuned, and 'higher than Glasgow-MRC') are computed against the MRC and Glasgow norms using GPT-4o-mini, whose training data are not disclosed. The manuscript itself identifies this exact threat, stating that 'good performance on that dataset cannot necessarily be generalized to other words' (Section 4) and recommending membership-inference audits or newly collected human ratings (Section 7.1). The case study performs no such audit and collects no new human data, so the reported numbers are equally compatible with memorization of the evaluation items. Please either add a contamination check (e.g., the membership-inference method cited in Section 7.1), use norms collected after the model's cutoff, or collect a fresh human validation sample, and rephrase the abstract and A.4.3 claims as provisional until such evidence is available.","section":"Abstract; Section 4; Section 7.1; A.4.3"},{"comment":"The fine-tuning gains of 0.08-0.10 are reported without confidence intervals, and the best of the three fine-tuning targets (Glasgow, MRC, mean) was selected after evaluating all three on the same 1045-word test set. This is a model-selection-on-the-test-set procedure, which inflates the expected gain. The same concern applies to prompt selection in Figure 2, where prompts are ranked on the full 2545-word set that is later split for fine-tuning. Please report bootstrap or other confidence intervals for the correlations and their differences, and either fix the fine-tuning target and prompt a priori or use a three-way split (train/selection/test) or nested cross-validation.","section":"A.4.1; A.4.3; Figure 3"},{"comment":"The claim that the fine-tuned model 'achieves a higher correlation than the one observed between Glasgow and MRC' compares correlations computed on different subsets. The Glasgow-MRC inter-correlation of r = .79 / rho = .80 is reported over the 2545 common words (Figure 2), while the fine-tuned correlations are reported over the 1045 held-out test words (Figure 3). Before making this comparison, the Glasgow-MRC inter-correlation must be recomputed on the same 1045 test words, with a confidence interval; otherwise the comparison is not apples-to-apples.","section":"Figure 2; Figure 3; A.3.4; A.4.3"},{"comment":"The logprobs weighting method is described as a weighted average of the top-k tokens, but the value of k, or a probability cutoff, is never specified. Since logprobs are credited with an improvement of up to .10 in correlation, this is a load-bearing parameter for reproducibility. Please report the exact top-k value or cutoff, and state whether this parameter was tuned on the validation data; if it was tuned, the resulting correlations should be interpreted with the same selection caveats as the prompt and fine-tuning choices.","section":"A.3.1; A.3.4; Figure 2"},{"comment":"The random split into 1500 training and 1045 test words is a single split, and the paper's statement that 'with sample sizes of more than 1000, the results are unlikely to differ much between different random splits' is offered without supporting evidence. The reported gains are small enough (0.08-0.10) that split-to-split variation could change the conclusions. Please report results across several random splits, or a bootstrap over splits, and in any case provide the variance or confidence intervals for the gains.","section":"A.4.1; A.4.3"}],"minor_comments":[{"comment":"There is a typo in the Introduction ('Thopmson & Lupyan, 2018'), and the reference list contains an inconsistent author name ('Want & Xu, 2024' versus 'Wang, T., & Xu, X.'); please make these consistent.","section":"Introduction; References"},{"comment":"The statement that humans can provide 'trustworthy ratings of word features at a rate of 1000 per hour' needs clarification; if this means 1000 individual item ratings per participant per hour it is implausible, and if it means something else (e.g., across a group of participants) the intended unit should be stated.","section":"A.1.1"},{"comment":"The few-shot anchors in the best prompt ('imam' = 2, 'theology' = 4, 'fridge' = 7) are introduced without any explanation of how these particular words and values were chosen; because anchor values are free parameters, the selection procedure should be disclosed.","section":"A.3.2; Figure 2"},{"comment":"The phrase 'as we can expect' after the result that fine-tuning on a database's own estimates gives the largest held-out correlation is unclear; if the claim is that this is a general expectation, it should be justified, otherwise it should be removed.","section":"A.4.3"},{"comment":"The suggested threshold of 0.96 for the two-permutation reliability check is presented without derivation or citation; a brief justification or reference would help readers calibrate this rule of thumb.","section":"Section 3"}],"recommendation":"major_revision","confidential_remarks":"The paper is publishable in principle, and the methodological guide plus framework are likely to be useful to the field. The main obstacle is the case study's evidential status: the headline 0.9 result is presented as a validation success, but the paper's own caveats about contamination and generalization apply directly to it. I would ask the editor to require the contamination audit or fresh human data, or a substantial weakening of the abstract and A.4.3 claims, before acceptance. I would also encourage the authors to ensure the GitHub and Zenodo artifacts are version-stamped at the time of acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Javier,\n\nQuick take: this is a solid practical guide with real artifacts — code, configs, data, and a step-by-step case study. If you're advising someone who wants to generate psycholinguistic norms with LLMs, this is a good starting point. The paper also does something rare: it documents dead ends and tells you when the method fails, not just when it works. The contamination warning in Section 7.1 is honest and specific.\n\nWhat's actually new: the framework (WordsGPT) that implements the pipeline, and the integration of prompt engineering, logprobs weighting, and fine-tuning into one workflow. The case study numbers themselves are not new in kind — the group has published similar results before — but the public framework and the candid comparison of prompt variants are useful contributions.\n\nWhere I'd be careful: the headline numbers sit on a shaky base. The fine-tuned GPT-4o-mini result of ~0.9 is measured against MRC and Glasgow familiarity norms, which are old, public, and plausibly in the model's training data. The paper itself says \"good performance on that dataset cannot necessarily be generalized to other words,\" and it lists data contamination as a central pitfall — but the case study performs no audit and collects no new human ratings. So the 0.9 and the \"better than Glasgow-MRC agreement\" claim should be read as illustrative, not as evidence that the model generalizes to new words. There's also a milder issue: the best of three fine-tuning targets was chosen after evaluating on the same 1045-word test set, and no confidence intervals are reported. The reported gains of 0.08-0.10 could easily shrink on a different split or a fresh sample.\n\nThat said, these problems don't undercut the paper's main value. The guide's recommendations — validate against human data, prefer open models when possible, treat LLM norms as augmentation not replacement, keep held-out data — are sensible and appropriately cautious. The framework and the accompanying documentation are reproducible in a way that matters.\n\nWho this is for: researchers considering LLM-based norming, and reviewers evaluating such work. It deserves serious peer review — not because the case study settles anything, but because the guide and framework will shape practice in the field.","headline":"A useful, candid practical guide to LLM-based norming, but the case study's headline correlations should not be read as evidence of generalization to new words.","tokens_in":30072,"tokens_out":1990,"would_cite":true,"duration_ms":17219,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned LLMs can produce word familiarity ratings that match human norms.","keywords":["LLMs","word features","fine-tuning","psycholinguistic norms","familiarity ratings","validation","prompt engineering","tutorial"],"falsifier":"Collect fresh human familiarity ratings for a few hundred words that appear in neither the MRC nor the Glasgow norms, run the best fine-tuned model on them, and compare correlations: if the Spearman correlation with these new ratings is substantially below the roughly 0.9 obtained on the existing datasets, or below the human-human inter-rater correlation, the claim that the norms generalize to new words is falsified.","tokens_in":1610,"feed_emoji":"🧠","tokens_out":4922,"duration_ms":85676,"temperature":0.7,"pith_summary":"This paper argues that large language models can become a practical addition to the psycholinguistic norming toolbox, provided that every use is validated against human “gold standard” ratings. In the English familiarity case study, base GPT-4o-mini prompts reached a Spearman correlation of about 0.8 with two established human norm sets, and fine-tuning on 1,500 human-rated words raised the best model to roughly 0.9—above the correlation between the two human datasets themselves. The authors deliver a step-by-step methodology, a software framework, and a list of pitfalls, with the central message that LLM-generated norms may augment human data but should not replace them. A sympathetic reader takes away a concrete recipe: collect or reuse human ratings, generate estimates with temperature zero and independent per-word prompts, use logprobs when possible, fine-tune on a training split, and validate on a held-out test split before scaling to new words.","feed_headline":"Fine-tuned LLM ratings match human word norms","feed_subtitle":"A validated workflow pushes Spearman correlations from 0.8 to 0.9, above the human-dataset baseline.","key_machinery":"The central mechanism is a five-step validation loop: obtain human gold-standard ratings for a few thousand words, select model, configuration, and prompt, generate estimates as independent per-word API calls at temperature 0, split the gold standard into training and test sets and fine-tune on the training portion, and expand to new words only after test-set performance is acceptable. Two specific devices carry much of the accuracy: using logprobs to compute a probability-weighted average over the model’s answer tokens rather than taking a single sampled output, and writing prompts that anchor the construct with definitions and extreme examples. The loop is what converts an inscrutable model into a measurable instrument: every claim about quality is checked against human ratings before the estimates are treated as norms.","core_discovery":"The paper’s central claim is that LLM estimates of word characteristics can be calibrated to human judgments tightly enough for practical use. The case study shows this directly: for English word familiarity, the best base-model configuration reaches Spearman correlations around 0.8 with the MRC and Glasgow human norms, and the best fine-tuned GPT-4o-mini model—trained on 1,500 of the 2,545 shared words and evaluated on the remaining 1,045—reaches roughly 0.9, correlating more strongly with each human dataset than the two human datasets correlate with each other. The paper frames this as evidence that LLM-generated norms can be a valid augmentation of human norming when the generation procedure is validated against human gold standards, and it advises against publishing LLM-based norms without such validation.","pith_inferences":["If the validation logic holds on freshly collected words, the same pipeline could be used to build norms for languages and populations where megastudy data are scarce, because the human gold standard can be collected once for a few thousand items and the model then extends the scale to hundreds of thousands of forms.","The fine-tuning gains may partly reflect benchmark contamination: models trained on public MRC or Glasgow ratings could inflate the validation correlation, so the strongest test is to compare fine-tuned models against newly collected ratings rather than reused datasets.","One natural extension the paper leaves implicit is to use the logprobs distribution itself as a measure of the model’s uncertainty per word, which could let researchers flag low-confidence estimates for targeted human checking.","The result that fine-tuning helps the weaker open-weight model much more than the commercial one suggests model-agnostic improvements may be larger as open models improve, potentially making the method independent of commercial APIs."],"forward_implications":["Researchers can generate psycholinguistic norms for tens or hundreds of thousands of words after validating on a few thousand human ratings, rather than collecting ratings for every word.","Fine-tuning on roughly 1,500–3,000 human ratings is enough to reach or exceed human-human agreement in the case study, with diminishing returns beyond roughly 2,000 Likert ratings and 5,000 reaction-time items.","The same pipeline can be applied to other constructs and languages, but performance varies, so a human gold standard is always required before publishing LLM-derived norms.","LLM-generated norms should be treated as augmentations of human datasets, not replacements; the paper recommends keeping human judgments in the loop.","Open-weight models that underperform out of the box can be pulled nearly up to commercial-model level by fine-tuning."],"supporting_citations":[{"why":"Supplies the MRC familiarity ratings that serve as one of the two human gold standards for validation.","marker":"Coltheart et al., 1981"},{"why":"Supplies the Glasgow Norms, the second human gold standard, and the familiarity definition and prompt wording used in the case study.","marker":"Scott et al., 2019"},{"why":"Introduces the logprobs-weighted estimation and model-evaluation approach that the methodology builds on.","marker":"Conde et al., 2025a"},{"why":"Demonstrates that fine-tuning on a few thousand human AoA ratings raises LLM correlations to human inter-rater level.","marker":"Sendín et al., 2025"},{"why":"Shows that fine-tuning on 3,000 lexical-decision response times can predict megastudy RTs for much larger word sets.","marker":"Martínez et al., 2025"},{"why":"Provides substitution analyses and systematic-bias checks that the paper recommends beyond raw correlation with human norms.","marker":"Trott, 2024a"}],"fun_headline_variants":["LLM norms rival human ratings in word familiarity","Fine-tuned LLMs outperform human consensus on word norms","Guide to LLM-based word norms with human validation","LLM ratings hit 0.9 correlation with human norms"],"cache_read_input_tokens":32128,"weakest_assumption_plain":"The method’s usefulness rests on the assumption that a model that matches human ratings on a few thousand already-published words will also match human ratings on new words, which requires that those published ratings were not baked into the model’s training and that the train/test split mirrors the broader lexicon.","fun_headline_variants_meta":{"raw":{"variants":["LLM norms rival human ratings in word familiarity","Fine-tuned LLMs outperform human consensus on word norms","Guide to LLM-based word norms with human validation","LLM ratings hit 0.9 correlation with human norms"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000612,"raw_usage":{"total_tokens":2877,"prompt_tokens":1006,"completion_tokens":1871,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":622,"completion_tokens_details":{"reasoning_tokens":1806}},"tokens_in":622,"tokens_out":1871,"duration_ms":12399,"temperature":1.0,"reasoning_tokens":1806,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:50:30.925394+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect fresh human familiarity ratings for a few hundred words that appear in neither the MRC nor the Glasgow norms, run the best fine-tuned model on them, and compare correlations: if the Spearman correlation with these new ratings is substantially below the roughly 0.9 obtained on the existing datasets, or below the human-human inter-rater correlation, the claim that the norms generalize to new words is falsified.","supporting_citations":[],"review_version":2}