Adding LLMs to the psycholinguistic norming toolbox: A practical guide to getting the most out of human ratings

Carlos Arriaga; Clarence Gerald Green; Gonzalo Mart\'inez; Javier Conde; Marc Brysbaert; Mar\'ia Grandury; Pedro Reviriego; Sean Trott; Tairan Fu; Thomas Clark

REVIEW 5 major objections 5 minor 5 references

Adding LLMs to the psycholinguistic norming toolbox: A practical guide to getting the most out of human ratings

T0 review · 5 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuned LLMs can produce word familiarity ratings that match human norms.

desk verdict A useful, candid practical guide to LLM-based norming, but the case study's headline correlations should not be read as evidence of generalization to new words. read the letter →

arxiv 2509.14405 v1 pith:QMR2MICQ submitted 2025-09-17 cs.CL

Javier Conde , María Grandury , Tairan Fu , Carlos Arriaga , Gonzalo Martínez , Thomas Clark , Sean Trott , Clarence Gerald Green

show 2 more authors

Pedro Reviriego Marc Brysbaert

This is my paper · ORCID

classification cs.CL

keywords LLMswordfeaturesfine-tuningpsycholinguisticnormsfamiliarityratingsvalidationpromptengineeringtutorial

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that large language models can become a practical addition to the psycholinguistic norming toolbox, provided that every use is validated against human “gold standard” ratings. In the English familiarity case study, base GPT-4o-mini prompts reached a Spearman correlation of about 0.8 with two established human norm sets, and fine-tuning on 1,500 human-rated words raised the best model to roughly 0.9—above the correlation between the two human datasets themselves. The authors deliver a step-by-step methodology, a software framework, and a list of pitfalls, with the central message that LLM-generated norms may augment human data but should not replace them. A sympathetic reader takes away a concrete recipe: collect or reuse human ratings, generate estimates with temperature zero and independent per-word prompts, use logprobs when possible, fine-tune on a training split, and validate on a held-out test split before scaling to new words.

What carries the argument

The central mechanism is a five-step validation loop: obtain human gold-standard ratings for a few thousand words, select model, configuration, and prompt, generate estimates as independent per-word API calls at temperature 0, split the gold standard into training and test sets and fine-tune on the training portion, and expand to new words only after test-set performance is acceptable. Two specific devices carry much of the accuracy: using logprobs to compute a probability-weighted average over the model’s answer tokens rather than taking a single sampled output, and writing prompts that anchor the construct with definitions and extreme examples. The loop is what converts an inscrutable model into a measurable instrument: every claim about quality is checked against human ratings before the estimates are treated as norms.

What would settle it

Collect fresh human familiarity ratings for a few hundred words that appear in neither the MRC nor the Glasgow norms, run the best fine-tuned model on them, and compare correlations: if the Spearman correlation with these new ratings is substantially below the roughly 0.9 obtained on the existing datasets, or below the human-human inter-rater correlation, the claim that the norms generalize to new words is falsified.

Watch

Extended reading notes

Core claim

The paper’s central claim is that LLM estimates of word characteristics can be calibrated to human judgments tightly enough for practical use. The case study shows this directly: for English word familiarity, the best base-model configuration reaches Spearman correlations around 0.8 with the MRC and Glasgow human norms, and the best fine-tuned GPT-4o-mini model—trained on 1,500 of the 2,545 shared words and evaluated on the remaining 1,045—reaches roughly 0.9, correlating more strongly with each human dataset than the two human datasets correlate with each other. The paper frames this as evidence that LLM-generated norms can be a valid augmentation of human norming when the generation procedure is validated against human gold standards, and it advises against publishing LLM-based norms without such validation.

Load-bearing premise

The method’s usefulness rests on the assumption that a model that matches human ratings on a few thousand already-published words will also match human ratings on new words, which requires that those published ratings were not baked into the model’s training and that the train/test split mirrors the broader lexicon.

Editorial extensions

If this is right

Researchers can generate psycholinguistic norms for tens or hundreds of thousands of words after validating on a few thousand human ratings, rather than collecting ratings for every word.
Fine-tuning on roughly 1,500–3,000 human ratings is enough to reach or exceed human-human agreement in the case study, with diminishing returns beyond roughly 2,000 Likert ratings and 5,000 reaction-time items.
The same pipeline can be applied to other constructs and languages, but performance varies, so a human gold standard is always required before publishing LLM-derived norms.
LLM-generated norms should be treated as augmentations of human datasets, not replacements; the paper recommends keeping human judgments in the loop.
Open-weight models that underperform out of the box can be pulled nearly up to commercial-model level by fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the validation logic holds on freshly collected words, the same pipeline could be used to build norms for languages and populations where megastudy data are scarce, because the human gold standard can be collected once for a few thousand items and the model then extends the scale to hundreds of thousands of forms.
The fine-tuning gains may partly reflect benchmark contamination: models trained on public MRC or Glasgow ratings could inflate the validation correlation, so the strongest test is to compare fine-tuned models against newly collected ratings rather than reused datasets.
One natural extension the paper leaves implicit is to use the logprobs distribution itself as a measure of the model’s uncertainty per word, which could let researchers flag low-confidence estimates for targeted human checking.
The result that fine-tuning helps the weaker open-weight model much more than the commercial one suggests model-agnostic improvements may be larger as open models improve, potentially making the method independent of commercial APIs.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Javier,

Quick take: this is a solid practical guide with real artifacts — code, configs, data, and a step-by-step case study. If you're advising someone who wants to generate psycholinguistic norms with LLMs, this is a good starting point. The paper also does something rare: it documents dead ends and tells you when the method fails, not just when it works. The contamination warning in Section 7.1 is honest and specific.

What's actually new: the framework (WordsGPT) that implements the pipeline, and the integration of prompt engineering, logprobs weighting, and fine-tuning into one workflow. The case study numbers themselves are not new in kind — the group has published similar results before — but the public framework and the candid comparison of prompt variants are useful contributions.

Where I'd be careful: the headline numbers sit on a shaky base. The fine-tuned GPT-4o-mini result of ~0.9 is measured against MRC and Glasgow familiarity norms, which are old, public, and plausibly in the model's training data. The paper itself says "good performance on that dataset cannot necessarily be generalized to other words," and it lists data contamination as a central pitfall — but the case study performs no audit and collects no new human ratings. So the 0.9 and the "better than Glasgow-MRC agreement" claim should be read as illustrative, not as evidence that the model generalizes to new words. There's also a milder issue: the best of three fine-tuning targets was chosen after evaluating on the same 1045-word test set, and no confidence intervals are reported. The reported gains of 0.08-0.10 could easily shrink on a different split or a fresh sample.

That said, these problems don't undercut the paper's main value. The guide's recommendations — validate against human data, prefer open models when possible, treat LLM norms as augmentation not replacement, keep held-out data — are sensible and appropriately cautious. The framework and the accompanying documentation are reproducible in a way that matters.

Who this is for: researchers considering LLM-based norming, and reviewers evaluating such work. It deserves serious peer review — not because the case study settles anything, but because the guide and framework will shape practice in the field.

Referee Report

5 major / 5 minor

Summary. The paper is a practical methodological guide to estimating word-level psycholinguistic norms with LLMs. It covers model selection, API versus web interfaces, prompt engineering, logprob-based aggregation, fine-tuning, validation against human norms, and a list of pitfalls and open questions. The authors also provide a Python framework and a case study on English familiarity in which base GPT-4o-mini estimates correlate at Spearman about 0.8 with MRC and Glasgow human norms, and fine-tuning increases this to about 0.9, with the best fine-tuned model reportedly exceeding the correlation between the Glasgow and MRC datasets themselves. The contribution is framed as a reference guide plus an illustrative workflow, not as a new full-scale norming database.

Significance. If the headline case-study numbers were trustworthy, this would be a genuinely useful contribution. The paper ships a reproducible framework with code and data, gives concrete and actionable advice on prompts, logprobs, fine-tuning, and validation, and is unusually candid about dead ends and limitations. The explicit emphasis on human gold-standard validation, inter-human baselines, and data contamination is a strength, and the case study is clearly described in enough detail to be replicated. The significance is contingent, however, on the validity of the 0.8/0.9 correlation claims, which currently rest on comparisons with old, widely available norms under an undisclosed training-data regime, with prompt and fine-tuning-target selection performed on the same evaluation sets.

major comments (5)

[Abstract; Section 4; Section 7.1; A.4.3] The headline correlations (0.8 base, 0.9 fine-tuned, and 'higher than Glasgow-MRC') are computed against the MRC and Glasgow norms using GPT-4o-mini, whose training data are not disclosed. The manuscript itself identifies this exact threat, stating that 'good performance on that dataset cannot necessarily be generalized to other words' (Section 4) and recommending membership-inference audits or newly collected human ratings (Section 7.1). The case study performs no such audit and collects no new human data, so the reported numbers are equally compatible with memorization of the evaluation items. Please either add a contamination check (e.g., the membership-inference method cited in Section 7.1), use norms collected after the model's cutoff, or collect a fresh human validation sample, and rephrase the abstract and A.4.3 claims as provisional until such evidence is available.
[A.4.1; A.4.3; Figure 3] The fine-tuning gains of 0.08-0.10 are reported without confidence intervals, and the best of the three fine-tuning targets (Glasgow, MRC, mean) was selected after evaluating all three on the same 1045-word test set. This is a model-selection-on-the-test-set procedure, which inflates the expected gain. The same concern applies to prompt selection in Figure 2, where prompts are ranked on the full 2545-word set that is later split for fine-tuning. Please report bootstrap or other confidence intervals for the correlations and their differences, and either fix the fine-tuning target and prompt a priori or use a three-way split (train/selection/test) or nested cross-validation.
[Figure 2; Figure 3; A.3.4; A.4.3] The claim that the fine-tuned model 'achieves a higher correlation than the one observed between Glasgow and MRC' compares correlations computed on different subsets. The Glasgow-MRC inter-correlation of r = .79 / rho = .80 is reported over the 2545 common words (Figure 2), while the fine-tuned correlations are reported over the 1045 held-out test words (Figure 3). Before making this comparison, the Glasgow-MRC inter-correlation must be recomputed on the same 1045 test words, with a confidence interval; otherwise the comparison is not apples-to-apples.
[A.3.1; A.3.4; Figure 2] The logprobs weighting method is described as a weighted average of the top-k tokens, but the value of k, or a probability cutoff, is never specified. Since logprobs are credited with an improvement of up to .10 in correlation, this is a load-bearing parameter for reproducibility. Please report the exact top-k value or cutoff, and state whether this parameter was tuned on the validation data; if it was tuned, the resulting correlations should be interpreted with the same selection caveats as the prompt and fine-tuning choices.
[A.4.1; A.4.3] The random split into 1500 training and 1045 test words is a single split, and the paper's statement that 'with sample sizes of more than 1000, the results are unlikely to differ much between different random splits' is offered without supporting evidence. The reported gains are small enough (0.08-0.10) that split-to-split variation could change the conclusions. Please report results across several random splits, or a bootstrap over splits, and in any case provide the variance or confidence intervals for the gains.

minor comments (5)

[Introduction; References] There is a typo in the Introduction ('Thopmson & Lupyan, 2018'), and the reference list contains an inconsistent author name ('Want & Xu, 2024' versus 'Wang, T., & Xu, X.'); please make these consistent.
[A.1.1] The statement that humans can provide 'trustworthy ratings of word features at a rate of 1000 per hour' needs clarification; if this means 1000 individual item ratings per participant per hour it is implausible, and if it means something else (e.g., across a group of participants) the intended unit should be stated.
[A.3.2; Figure 2] The few-shot anchors in the best prompt ('imam' = 2, 'theology' = 4, 'fridge' = 7) are introduced without any explanation of how these particular words and values were chosen; because anchor values are free parameters, the selection procedure should be disclosed.
[A.4.3] The phrase 'as we can expect' after the result that fine-tuning on a database's own estimates gives the largest held-out correlation is unclear; if the claim is that this is a general expectation, it should be justified, otherwise it should be removed.
[Section 3] The suggested threshold of 0.96 for the two-permutation reliability check is presented without derivation or citation; a brief justification or reference would help readers calibrate this rule of thumb.

Circularity Check

1 steps flagged · score 3.0 of 10

Case-study gain is partly selection-optimized: the prompt and fine-tuning target were chosen on the same 2545-word Glasgow/MRC pool later split to form the 1045-word test set, so the reported 0.9 correlation is not a fully independent held-out prediction; core validation against external human norms is otherwise non-circular.

fitted input called prediction [A.3.4 (Figure 2) and A.4.3 (Training and evaluation)]
"Figure 2 presents some results for the prompts we discussed with respect to English familiarity. The data are based on the 2545 words for which there are human familiarity ratings both in the MRC database (Coltheart et al., 1981) and Glasgow database (Scott et al., 2019). ... Following the English familiarity case study, we decided to split the 2545-item dataset into a training set for fine-tuning (1500 entries) and a test set for validation (1045 entries)."

The winning prompt configuration was selected by ranking correlations on the full 2545-word Glasgow/MRC pool (Figure 2). The fine-tuning evaluation then reports correlations on a 1045-word test set randomly drawn from that same 2545-word pool. Because those test labels were already used to choose the prompt, the test-set correlation is not a clean held-out prediction; it is inflated by selection on the evaluation data. The same issue applies to the choice among the three fine-tuning targets (Glasgow, MRC, and their mean), where the 'best' model and the claim that it 'achieves a higher correlation than the one observed between Glasgow and MRC' are selected after inspecting test-set correlations.

full rationale

The core evaluation is not circular in the strongest sense: LLM outputs for the held-out test words are compared against external human norms (MRC and Glasgow), and the fine-tuning test words are not in the fine-tuning training set. The abstract's headline correlations and the 'higher than Glasgow-MRC' claim are therefore genuine empirical comparisons, not identities with the model's inputs. However, the case study's prompt engineering and fine-tuning-target selection were performed on the same 2545-word Glasgow/MRC dataset later used to define the 1045-word test set. The paper explicitly recommends preserving held-out data to avoid this ('researchers can ameliorate this risk by preserving some amount of held-out data' in A.3.2; 'it is a good idea to always keep one data sample until the very end of your pipeline' in A.4.1), but the illustrative case study does not follow that advice for prompt selection or target selection. This makes the reported fine-tuning gains partially selection-optimized rather than fully out-of-sample. Data contamination is also acknowledged as a limitation, and the case study performs no membership-inference audit, but contamination is an external-validity threat rather than a circularity of derivation. Self-citations in the paper are used as reports of the authors' experience and prior studies, not as a load-bearing uniqueness argument or as a substitute for the empirical validation, so they do not constitute circularity. Overall, the methodology's central recommendation—validate LLM estimates against human gold standards—is independent and sound; the circularity concern is limited to the way the illustrative numbers were selected.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new theoretical entities are postulated. The case study depends on hand-chosen prompt examples and split choices, which are free parameters in the demonstration. The axiomatic burden is the standard assumption of correlational validation against human norms, plus the generalizability and contamination caveats the paper itself emphasizes.

free parameters (3)

few-shot anchor ratings in best prompt = imam=2, theology=4, fridge=7
Hand-picked by the authors and selected because the few-shot prompt performed best on the validation set (A.3.2, Figure 2). This is a tuned choice rather than a fixed input.
fine-tuning train/test split = 1,500 training / 1,045 test (random_state 42)
The split size is a pragmatic choice; the paper recommends at least 3,000 words but uses the 2,545 available words. Different splits could change the reported gains.
logprobs weighted average cutoff = integer tokens 0-999
The framework uses logprobs only when the output is limited to an integer between 0 and 999 (A.3.1). This modeling choice affects the resulting estimates and is not independently derived.

assumptions (4)

domain assumption Human ratings from MRC and Glasgow are valid gold standards for familiarity, and their inter-dataset correlation (r=.79, ρ=.80) represents a realistic performance ceiling.
Used in A.1.1 and A.3.4 as the baseline against which LLM estimates are judged.
domain assumption LLM outputs can be elicited as single-number ratings and compared with human averages via Spearman and Pearson correlation.
The whole methodology rests on this comparability, discussed in Sections 4 and 5.
domain assumption Correlation with a human dataset is a valid measure of the utility of LLM-generated norms.
The paper recommends correlation as the main validation metric (Section 4, A.3.4).
domain assumption Data contamination in commercial models cannot be fully ruled out, so validation on new or unlikely-to-be-trained data is required.
Stated as a pitfall in Section 7.1; the case study itself uses two older datasets (MRC 1981, Glasgow 2019) that may or may not be in training data.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Adding LLMs to the psycholinguistic norming toolbox: A practical guide to getting the most out of human ratings." pith.science (2026). https://pith.science/paper/QMR2MICQ

@misc{pith2026250914405,
  author       = {Pith},
  title        = {Pith review of: Adding LLMs to the psycholinguistic norming toolbox: A practical guide to getting the most out of human ratings},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QMR2MICQ}},
  note         = {Machine review of arXiv:2509.14405}
}

read the original abstract

Word-level psycholinguistic norms lend empirical support to theories of language processing. However, obtaining such human-based measures is not always feasible or straightforward. One promising approach is to augment human norming datasets by using Large Language Models (LLMs) to predict these characteristics directly, a practice that is rapidly gaining popularity in psycholinguistics and cognitive science. However, the novelty of this approach (and the relative inscrutability of LLMs) necessitates the adoption of rigorous methodologies that guide researchers through this process, present the range of possible approaches, and clarify limitations that are not immediately apparent, but may, in some cases, render the use of LLMs impractical. In this work, we present a comprehensive methodology for estimating word characteristics with LLMs, enriched with practical advice and lessons learned from our own experience. Our approach covers both the direct use of base LLMs and the fine-tuning of models, an alternative that can yield substantial performance gains in certain scenarios. A major emphasis in the guide is the validation of LLM-generated data with human "gold standard" norms. We also present a software framework that implements our methodology and supports both commercial and open-weight models. We illustrate the proposed approach with a case study on estimating word familiarity in English. Using base models, we achieved a Spearman correlation of 0.8 with human ratings, which increased to 0.9 when employing fine-tuned models. This methodology, framework, and set of best practices aim to serve as a reference for future research on leveraging LLMs for psycholinguistic and lexical studies.

Figures

Figures reproduced from arXiv: 2509.14405 by the authors.

**Figure 2.** Spearman correlations of the GPT and Llama prompts with Glasgow and MRC English familiarity databases (over the 2545 common words). Above the diagonal: correlations calculated with logprobs (when possible); below the diagonal: correlation using the output of the model. A.4.Fine-tuning the model Depending on the quality of the LLM estimates, you may look into improving them by fine-tuning the model. In fine-tuning, t… view at source ↗

**Figure 3.** Correlations of the GPT and Llama fine-tuned models with Glasgow and MRC English familiarity databases, v02_logprobs and v09_logprobs prompt versions without fine-tuning, over the 1045 validation words. Above the diagonal: Pearson correlations; below the diagonal: Spearman correlation. A.5.Data Augmentation Once you are happy with the outcome of the (fine-tuned) LLM you are using, you can extend the estimates to all… view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

5 extracted references · 3 canonical work pages

[1]

J., Trager, J., Park, P

Abdurahman, S., Atari, M., Karimi -Malekabadi, F., Xue, M. J., Trager, J., Park, P. S., ... & Dehghani, M. (2024). Perils and opportunities in using large language models in psychological research. PNAS nexus, 3(7), pgae245. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., ... & McGrew, B. (2023). Gpt-4 technical report. arXiv pre...

arXiv 2024
[5]

Messeri, L., & Crockett, M. J. (2024). Artificial intelligence and illusions of understanding in scientific research. Nature, 627(8002), 49-58. Moreno-Martínez, F. J., Montoro, P. R., & Rodríguez -Rojo, I. C. (2014). Spanish norms for age of acquisition, concept familiarity, lexical frequency, manipulability, typicality, and other variables for 820 words ...

arXiv 2024
[1239]

(2024 a)

Trott, S. (2024 a). Can large language models help augment English psycholinguistic datasets?. Behavior Research Methods, 56(6), 6082-6100. Trott, S. (2024b). Large language models and the wisdom of small crowds. Open Mind, 8, 723-738. Turpin, M., Michael, J., Perez, E., & Bowman, S. (2023). Language models don't always say what they think: Unfaithful exp...

work page 2024
[1270]

A., & Brysbaert, M

Sendín, E., Conde, J., Reviriego , P., Haro, J., Ferré, P., Hinojosa, J. A., & Brysbaert, M. (2025). Combining the power of large language models with finetuning based on strategically collected human ratings: A case study about age -of-acquisition estimates of Spanish words. Preprint. 10.13140/RG.2.2.27255.12967 Solovyev, V., Islamov, M., & Bayrasheva, V...

work page arXiv 2025
[2407]

Learning and Evaluating Emotion Lexicons for 91 Languages

Botarleanu, R. M., Watanabe, M., Dascalu, M., Crossley, S. A., & McNamara, D. S. (2024). Multilingual Age of Exposure 2.0. International Journal of Artificial Intelligence in Education, 34(4), 1353-1377. Bricken, et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread. Brysbaert, M., Martín...

work page Pith review arXiv 2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

J., Trager, J., Park, P

Abdurahman, S., Atari, M., Karimi -Malekabadi, F., Xue, M. J., Trager, J., Park, P. S., ... & Dehghani, M. (2024). Perils and opportunities in using large language models in psychological research. PNAS nexus, 3(7), pgae245. Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., ... & McGrew, B. (2023). Gpt-4 technical report. arXiv pre...

arXiv 2024

[2] [5]

Messeri, L., & Crockett, M. J. (2024). Artificial intelligence and illusions of understanding in scientific research. Nature, 627(8002), 49-58. Moreno-Martínez, F. J., Montoro, P. R., & Rodríguez -Rojo, I. C. (2014). Spanish norms for age of acquisition, concept familiarity, lexical frequency, manipulability, typicality, and other variables for 820 words ...

arXiv 2024

[3] [1239]

(2024 a)

Trott, S. (2024 a). Can large language models help augment English psycholinguistic datasets?. Behavior Research Methods, 56(6), 6082-6100. Trott, S. (2024b). Large language models and the wisdom of small crowds. Open Mind, 8, 723-738. Turpin, M., Michael, J., Perez, E., & Bowman, S. (2023). Language models don't always say what they think: Unfaithful exp...

work page 2024

[4] [1270]

A., & Brysbaert, M

Sendín, E., Conde, J., Reviriego , P., Haro, J., Ferré, P., Hinojosa, J. A., & Brysbaert, M. (2025). Combining the power of large language models with finetuning based on strategically collected human ratings: A case study about age -of-acquisition estimates of Spanish words. Preprint. 10.13140/RG.2.2.27255.12967 Solovyev, V., Islamov, M., & Bayrasheva, V...

work page arXiv 2025

[5] [2407]

Learning and Evaluating Emotion Lexicons for 91 Languages

Botarleanu, R. M., Watanabe, M., Dascalu, M., Crossley, S. A., & McNamara, D. S. (2024). Multilingual Age of Exposure 2.0. International Journal of Artificial Intelligence in Education, 34(4), 1353-1377. Bricken, et al. (2023). Towards Monosemanticity: Decomposing Language Models With Dictionary Learning. Transformer Circuits Thread. Brysbaert, M., Martín...

work page Pith review arXiv 2024