Humans are more gullible than LLMs in believing common psychological myths

Bevan Koopman; Guido Zuccon

REVIEW 4 major objections 6 minor 15 references

Humans are more gullible than LLMs in believing common psychological myths

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLMs asked neutrally believe 8–24% of 50 common psychology myths, versus 51–63% for human students; retrieval-augmented generation lowers belief further, while biased prompts can raise it to 65%.

desk verdict A clean, useful benchmark result that is probably right in direction but not yet in magnitude, because the human baseline is unmatched and the LLM numbers have no error bars. read the letter →

arxiv 2507.12296 v1 pith:ZECF26NZ submitted 2025-07-16 cs.HC

Bevan Koopman , Guido Zuccon This is my paper

classification cs.HC

keywords psychologicalmythslargelanguagemodelsmythbeliefretrieval-augmentedgenerationpromptsensitivityMachinePsychologycognitivebiaseshuman-LLMcomparison

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large language models reproduce the human tendency to believe widely debunked claims in psychology, and whether that tendency can be reduced. Using a fixed list of 50 popular myths, it finds that when the models are asked neutrally whether each statement is true, they endorse only 8 to 24 percent of the myths, while junior and senior psychology students in the comparison baseline endorsed 63 and 51 percent. It then shows that feeding retrieved web passages into the prompt (retrieval-augmented generation, RAG) generally lowers belief further, in one model to zero, and that prompts expressing a user's prior belief can raise belief substantially, up to 65 percent. The authors take this as evidence that these LLMs are less gullible than humans on this material, that they internally contain refuting knowledge that targeted prompting can expose, and that RAG is a practical mitigation against myth belief and prompt-induced bias.

What carries the argument

The machinery is a fixed 50-item myth inventory whose ground truth is always False, combined with a binary belief measurement: a model answer of True counts as believing a myth. Onto that measurement the paper layers four conditions: a neutral prompt; two bias-inducing user prompts ('I'm very sceptical of ...' and 'I truly believe ...'); a RAG pipeline that retrieves the top passage from a general web corpus with BM25 and inserts it into the prompt; and two 'sway' prompts that first ask the model to argue for or against a myth and then feed that generated passage back into the RAG prompt. The human comparison is the published belief data from a 2024 study of 150 psychology students, so the whole argument is a comparison of the fraction of True responses across models, prompt types, and conditions.

What would settle it

Run the same 50 myths with exactly the same binary True/False wording the LLMs saw on a fresh sample of comparable human participants; if those humans also believe fewer than roughly a quarter of the myths, the claimed human–LLM gap disappears.

Watch

Extended reading notes

Core claim

Under a neutral, two-choice instruction to answer True or False, the four tested LLMs believe far fewer of the 50 popular psychology myths than the human comparison groups: Gemini-2.5-Flash and Llama-4-17B believe 8 percent, Llama-3.3-70B believes 22 percent, and GPT-4o believes 24 percent, versus 63 percent for junior and 51 percent for senior psychology students. Adding RAG with a general web corpus reduces belief in most models (Llama 4 goes to zero, GPT-4o to 16 percent), though it increases Gemini's belief from 8 to 10 percent. Biased user prompts also move belief in the expected directions: 'very sceptical' framing lowers it, 'I truly believe' framing raises it, and a 'Believer' sway prompt that asks the model to write a supporting passage before answering pushes belief as high as 65 percent for Llama 3. A statistical significance matrix reported in the appendix marks most human–LLM differences as significant. The authors conclude that LLMs do not mimic human myth-belief patterns, that their parameters encode both refuting and supporting material, and that retrieval grounding is a viable debiasing strategy.

Load-bearing premise

The headline comparison assumes that the published human belief rates, collected from psychology students in a survey, are directly comparable to the LLMs' binary True/False answers to a chat prompt.

Editorial extensions

If this is right

In neutral, unprimed interactions, these LLM families will usually answer that a popular psychology claim is false rather than endorsing it.
A user who frames a question with the belief that a myth is true can push several of these models to affirm the myth, so chatbot answers can reinforce rather than correct user bias.
Grounding the prompt in retrieved passages reduces both myth belief and sensitivity to biased framing, making RAG a practical safeguard for myth-prone topics.
The sway experiments show that the same model can be induced to produce both strong refutations and strong endorsements of the same myth, confirming that both viewpoints are encoded in its parameters.
The divergence from human belief rates counts as evidence for the emerging Machine Psychology view that LLMs do not automatically reproduce every human cognitive bias.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Beyond the paper, the same protocol could be run on general-population adults with exactly the same binary True/False wording as the LLMs; because the human baselines came from psychology students in a survey, a same-format replication would settle how much of the reported gap is an artifact of measurement differences.
The RAG debiasing effect suggests a general recipe for other misinformation domains: retrieving a neutral document before answering reduces reliance on biased framing and on the model's own priors; this is testable on health claims or political misinformation.
The 'Mixed Belief' responses, where a model says a myth might be true but lacks scientific evidence, are a natural probe for how LLMs balance what the prompt supplies with what they learned during training; classifying those responses could reveal a hidden gradient of conviction rather than a binary belief.
Because the tested models are general-purpose, safety-tuned systems, the low belief rate could be a byproduct of guardrails and training-data curation rather than emergent skepticism; a controlled pretraining study with a fully open model and myth-heavy versus myth-refuting corpora would separate those explanations.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: this is a plausible, clearly written benchmark study. The direction—LLMs endorse fewer common psychology myths than students do—is consistent across four models and several prompt variants. But the headline percentage gap (8-24% vs. 63/51%) deserves more trust than the current methods can support, because the human baseline comes from a different study with a different response format and the LLM numbers have no error bars.

What's actually new: applying Lilienfeld's 50-great-myths list to LLMs is a straightforward extension that hasn't been done before with this benchmark. The RAG mitigation result—belief generally drops, though not for every model—and the Sway conditions give a quick look at how contextual knowledge interacts with parametric knowledge. The paper is clear, the figure is legible, and the limitations section acknowledges several relevant caveats.

Soft spots: The human-LLM comparison is the load-bearing claim and it rests on an unverified assumption that Meinz et al.'s survey used the same binary response format as the LLM prompt. The paper says previous human studies 'simply asked if they believed each myth' but doesn't report whether that was a forced true/false, a Likert scale, or something with a 'don't know' option. A matched human run with the exact same prompt would settle this. Also, the LLM runs are single-shot with no repeated sampling or temperature settings, so there are no confidence intervals around the 8-24% figures. There's also a selection tweak in the Sway Believer condition: for 8 myths the authors manually altered the prompt until the model produced a supporting passage. That could bias the high belief rates they report for that condition. And no code or data are released, which hurts reproducibility for a benchmark this size.

Bottom line: the paper deserves a serious referee. I'd ask for a matched human control with the same binary question, repeated LLM runs with error bars, and release of the exact prompts and outputs. The direction is probably right, but the title as it stands is stronger than the evidence.

Referee Report

4 major / 6 minor

Summary. The paper evaluates whether LLMs believe common psychological myths from Lilienfeld et al. (2009). Using 50 myths, it prompts four LLMs (Llama-3.3-70B, Llama-4-17B, GPT-4o, Gemini-2.5-Flash) with neutral, sceptical, and believing prompts, and with RAG using BM25 top-1 passages from MSMARCO. The central claim (RQ1) is that LLMs under a neutral prompt exhibit myth belief rates of 8-24%, far below human psychology students' 63% (junior) and 51% (senior) from Meinz et al. (2024). RAG generally reduces belief, and sway prompts influence responses (RQ2 and RQ3). The paper also reports a chi-square significance matrix in Appendix B and includes a limitations section.

Significance. The study is a timely empirical benchmark in Machine Psychology. Its strengths are the use of a canonical myth list, multiple LLMs, and a transparent prompting and RAG pipeline. If the human baseline is comparably measured, the finding that LLMs endorse fewer myths than human students would be a meaningful data point. However, the headline quantitative comparison depends on an external, unmatched human baseline, and the statistical support is currently under-specified; these issues must be resolved before the claim can be accepted.

major comments (4)

[3.1, 4.1] The central comparison relies on an external human baseline whose response format is not reported. The LLM prompt (Table 1) forces a binary 'True / False' answer with an explanation, while the text only asserts that 'previous studies with human subjects simply asked if they believed each myth statement.' It is not stated whether Meinz et al. (2024) used a binary forced-choice, a Likert scale, an agree/disagree cutoff, or allowed a 'don't know' option, nor how the 63% and 51% figures were aggregated. A matched human measurement using the same item wording, response format, and scoring rule is needed to support the quantitative gap claimed in the abstract and Section 4.1. Without this, the 8-24% versus 51-63% comparison is not directly interpretable.
[Appendix B, 4.1] The statistical significance tests in Table 3 are not adequately defined to support claims of significance. The unit of analysis is unclear (50 myths versus 150 human participants versus repeated LLM runs), sample sizes are absent, and no correction for multiple comparisons is described. Additionally, the LLM results are single binary outcomes per myth with no repeated runs, so the reported proportions (e.g., 8%) have no variance; a chi-square test comparing a single 50-item binary vector to an aggregate human percentage is not a statistically valid procedure as presented. The paper should report a pre-registered analysis plan, confidence intervals, or repeated runs to substantiate the word 'significantly' in Section 4.1.
[3.3, RQ2] The claim that RAG mitigates myth belief is not accompanied by any evaluation of retrieval quality. The paper uses top-1 BM25 passages from MSMARCO but does not report whether these passages actually mention or debunk the myth. Without relevance judgments or manual inspection of the retrieved passages, the observed reduction in myth belief cannot be attributed to retrieving corrective information; it may be due to the changed prompt template or to irrelevant retrieved content. The authors should report retrieval relevance statistics or perform a controlled comparison against a non-informative retrieval baseline.
[3.2, 4.1] The experimental protocol for the LLM runs is underspecified. No information is given about the number of runs per prompt, temperature setting, random seed, or how often the model produced a 'Mixed Belief' response and how those were handled in the percentages. Given that LLM sampling is stochastic, the binary per-myth outcomes may vary across runs; the robustness of the aggregate rates (e.g., 8% for Llama-4 versus 22% for Llama-3.3) should be assessed with repeated sampling and confidence intervals.

minor comments (6)

[Throughout] The term 'Truely believe' appears in Table 1, Figure 1, and Section 3.2; it should be 'Truly believe'.
[Table 1] The Sceptic prompt reads 'Tell my why <myth> is a myth.' This appears to be a typo for 'Tell me why' and should be corrected.
[Figure 1 caption] The caption states 'Percentages are Believed Myth on Neural prompt.' The word 'Neural' should be 'Neutral'.
[Abstract, Section 4.1] The claims about 'humans' in the title and abstract overgeneralize; the human reference group is a specific sample of psychology students from one study, and the wording should be scoped accordingly (e.g., 'human psychology students in Meinz et al.').
[Section 2] The paper says the human judgements from Meinz et al. (2024) 'were made public,' but no repository link or data identifier is provided; this should be supplied for reproducibility.
[Appendix B] In Table 3, row E (GPT-4o) shows no asterisks in any column, which appears inconsistent with the surrounding pattern; this should be explained or the table should be corrected.

Circularity Check

1 steps flagged · score 2.0 of 10

No load-bearing circularity; the central human-vs-LLM claim rests on external benchmarks. One minor self-feedback artifact in the Sway Believer condition.

other [Section 3.4, 'Swaying Myth Beliefs with RAG', Table 1]
"To achieve this, we first prompt an LLM with the 'Sway' prompts of Table 1. The response is then treated as a RAG document and fed to the RAG prompt."

In the Sway Believer setting, the LLM is prompted to 'Write me a short passage explaining why my statement is true. Do not mention information about why it might be false or a myth.' That generated one-sided passage is then inserted as the RAG document for the same model to judge. The high belief rates (65%, 51%, 41%) therefore reflect the model endorsing its own generated text, not an independent evaluation of the myth. The outcome is largely a round-trip of the model's output, making this condition self-referential by construction. It is not load-bearing for the paper's headline RQ1 human-vs-LLM claim, which relies on the neutral-prompt measurements and the external Meinz et al. baseline.

full rationale

RQ1's human-vs-LLM comparison is an empirical benchmark, not a derivation. The human rates (63% and 51%) come from Meinz et al. (2024), an external published dataset; the LLM True/False rates are new measurements under a fixed system prompt; the 50 myths and their 'false' ground truth come from Lilienfeld et al. (2009). No parameter is fitted to the human baseline and then reported as a prediction, and no equation in the paper reduces to its own input. The RAG and Sway conditions are interventions, not derivations. The only self-referential element is the Sway Believer condition, where the model's own generated myth-supporting passage becomes the RAG document; that condition's high belief rates are a self-feedback artifact rather than an independent measurement. Because this condition is transparently described, secondary, and not used to support the headline RQ1 conclusion, the overall circularity is minimal. Score 2 reflects the minor self-feedback step, not load-bearing circularity.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities or fitted parameters. The main load-bearing inputs are the myth list, the human baseline, and the RAG corpus. The free-parameter ledger is light because the study is empirical and comparative, not generative.

free parameters (1)

Retrieval top-k = 1
The number of passages retrieved in the RAG pipeline was fixed at k=1. This is a design choice, not fitted to data, but the paper does not ablate it. Results could differ with larger k or different ranking.

assumptions (3)

domain assumption The 50 statements from Lilienfeld et al. (2009) are all myths, i.e., false.
Used as ground truth for scoring LLM answers. This is taken from the psychology literature without independent verification that every statement is unequivocally false in the LLM evaluation context. Some myths, such as 'Happiness mostly comes from our external circumstances', may be debatable rather than strictly false.
domain assumption The human baseline from Meinz et al. (2024) is comparable to LLM responses under the neutral prompt.
The paper compares LLM True/False answers to aggregate human belief rates from a prior study. This assumes the response format and participant pool do not substantially alter the measured belief rates.
domain assumption MSMARCO web passages are a representative sample of the general web content users would retrieve.
The RAG mitigation uses a single prebuilt MSMARCO passage index. The paper acknowledges in the limitations that different corpora (e.g., Wikipedia vs. social media) could change results, so this generalization is untested.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Humans are more gullible than LLMs in believing common psychological myths." pith.science (2026). https://pith.science/paper/ZECF26NZ

@misc{pith2026250712296,
  author       = {Pith},
  title        = {Pith review of: Humans are more gullible than LLMs in believing common psychological myths},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZECF26NZ}},
  note         = {Machine review of arXiv:2507.12296}
}

read the original abstract

Despite widespread debunking, many psychological myths remain deeply entrenched. This paper investigates whether Large Language Models (LLMs) mimic human behaviour of myth belief and explores methods to mitigate such tendencies. Using 50 popular psychological myths, we evaluate myth belief across multiple LLMs under different prompting strategies, including retrieval-augmented generation and swaying prompts. Results show that LLMs exhibit significantly lower myth belief rates than humans, though user prompting can influence responses. RAG proves effective in reducing myth belief and reveals latent debiasing potential within LLMs. Our findings contribute to the emerging field of Machine Psychology and highlight how cognitive science methods can inform the evaluation and development of LLM-based systems.

Figures

Figures reproduced from arXiv: 2507.12296 by the authors.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

15 extracted references · 10 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Adrian Furnham and David J. Hughes. 2014. https://doi.org/10.1177/0098628314537984 Myths and misconceptions in popular psychology: Comparing psychology students and the general public . Teaching of Psychology, 41(3):256--261

work page doi:10.1177/0098628314537984 2014
[4]

Thilo Hagendorff, Ishita Dasgupta, Marcel Binz, Stephanie C. Y. Chan, Andrew Lampinen, Jane X. Wang, Zeynep Akata, and Eric Schulz. 2024. https://arxiv.org/abs/2303.13988 Machine psychology . Preprint, arXiv:2303.13988

arXiv 2024
[5]

Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. 2023. https://doi.org/10.1038/s43588-023-00527-x Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt . Nature Computational Science, 3(10):833--838

doi:10.1038/s43588-023-00527-x 2023
[6]

Erik Jones and Jacob Steinhardt. 2022. Capturing failures of large language models via human cognitive biases. Advances in Neural Information Processing Systems, 35:11785--11799

work page 2022
[7]

Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2024. Language models, like humans, show content effects on reasoning tasks. PNAS, 3(7)

work page 2024
[8]

Scott O Lilienfeld, Steven Jay Lynn, John Ruscio, and Barry L Beyerstein. 2009. 50 great myths of popular psychology: Shattering widespread misconceptions about human behavior. John Wiley & Sons

work page 2009

Show all 15 references

[9]

Olivia Macmillan-Scott and Mirco Musolesi. 2024. (Ir)rationality and cognitive biases in large language models. Royal Society Open Science, 11(6):240255

2024
[10]

Elizabeth J Meinz, Jennifer L Tennison, and Whitney A Dominguez. 2024. Who believes the ``50 great myths of psychology''? Teaching of Psychology, 51(1):30--38

2024
[11]

Team OLMo, Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, Akshita Bhagia, Yuling Gu, Shengyi Huang, Matt Jordan, Nathan Lambert, Dustin Schwenk, Oyvind Tafjord, Taira Anderson, David Atkinson, Faeze Brahman, Christopher Clark, Pradeep Dasigi, Nouha Dziri, an...

2025 arXiv
[12]

Andreas Opedal, Alessandro Stolfo, Haruki Shirakami, Ying Jiao, Ryan Cotterell, Bernhard Sch \"o lkopf, Abulhair Saparov, and Mrinmaya Sachan. 2023. Do language models exhibit the same cognitive biases in problem solving as human learners? In International Conference on Machin...

2023
[13]

Jonathan Shaki, Sarit Kraus, and Michael Wooldridge. 2023. Cognitive effects in large language models. In ECAI 2023, pages 2105--2112. IOS Press

2023
[14]

Yasuaki Sumita, Koh Takeuchi, and Hisashi Kashima. 2025. Cognitive biases in large language models: A survey and mitigation experiments. In Proceedings of the 40th ACM/SIGAPP Symposium on Applied Computing, pages 1009--1011

2025
[15]

Gaurav Suri, Lily R Slater, Ali Ziaee, and Morgan Nguyen. 2024. Do large language models show decision heuristics similar to humans? a case study using gpt-3.5. Journal of Experimental Psychology: General, 153(4):1066

2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Adrian Furnham and David J. Hughes. 2014. https://doi.org/10.1177/0098628314537984 Myths and misconceptions in popular psychology: Comparing psychology students and the general public . Teaching of Psychology, 41(3):256--261

work page doi:10.1177/0098628314537984 2014

[4] [4]

Thilo Hagendorff, Ishita Dasgupta, Marcel Binz, Stephanie C. Y. Chan, Andrew Lampinen, Jane X. Wang, Zeynep Akata, and Eric Schulz. 2024. https://arxiv.org/abs/2303.13988 Machine psychology . Preprint, arXiv:2303.13988

arXiv 2024

[5] [5]

Thilo Hagendorff, Sarah Fabi, and Michal Kosinski. 2023. https://doi.org/10.1038/s43588-023-00527-x Human-like intuitive behavior and reasoning biases emerged in large language models but disappeared in chatgpt . Nature Computational Science, 3(10):833--838

doi:10.1038/s43588-023-00527-x 2023

[6] [6]

Erik Jones and Jacob Steinhardt. 2022. Capturing failures of large language models via human cognitive biases. Advances in Neural Information Processing Systems, 35:11785--11799

work page 2022

[7] [7]

Andrew K Lampinen, Ishita Dasgupta, Stephanie CY Chan, Hannah R Sheahan, Antonia Creswell, Dharshan Kumaran, James L McClelland, and Felix Hill. 2024. Language models, like humans, show content effects on reasoning tasks. PNAS, 3(7)

work page 2024

[8] [8]

Scott O Lilienfeld, Steven Jay Lynn, John Ruscio, and Barry L Beyerstein. 2009. 50 great myths of popular psychology: Shattering widespread misconceptions about human behavior. John Wiley & Sons

work page 2009