Could you be wrong: Debiasing LLMs using a metacognitive prompt for improving human decision making

Thomas T. Hills

REVIEW 3 major objections 6 minor 35 references

Could you be wrong: Debiasing LLMs using a metacognitive prompt for improving human decision making

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Asking an LLM 'could you be wrong?' makes it reveal its own biases and omitted evidence.

desk verdict The paper shows a simple metacognitive prompt can elicit self-critical LLM commentary, but it does not demonstrate debiasing; as a short demonstration it deserves a serious referee with a re-scope or added evidence. read the letter →

arxiv 2507.10124 v1 pith:N26QIUDJ submitted 2025-07-14 cs.AI

Thomas T. Hills This is my paper

classification cs.AI

keywords largelanguagemodelsmetacognitivepromptingdebiasingimplicitbiasepistemichumilityself-critiquechoiceoverloadpromptengineering

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a technique used to debias human decision makers—prompting people to consider why their judgment might be wrong—also works on large language models. The author applies the follow-up prompt 'could you be wrong?' to three known LLM failures: implicit discriminatory associations, inability to admit unanswerable medical questions, and confident summaries that omit counter-evidence. In each case, the prompt elicits a second response that names the bias, acknowledges the missing information, or surfaces the contradictory evidence that the first answer concealed. The larger claim is that human psychology offers a general, language-based strategy for debiasing LLMs that could remain useful as models change and specific biases shift.

What carries the argument

The central object is a metacognitive prompt, defined as a wording designed to make a decision maker reflect on the limits of their own thinking. The specific instance is 'could you be wrong?', appended after the model has produced an answer. The machinery is the model's own autoregressive generation: because output must exist before it can be evaluated, the first answer provides a concrete target, and the follow-up prompt invites the model to produce an adversarial crowd within itself—counterarguments, contradictory evidence, alternative interpretations, and admissions of bias. The mechanism the paper proposes is retrieval: LLMs trained on vast human text contain information about biases and counterevidence, but that information is only brought into the response when the prompt directs attention to it.

What would settle it

Run the same initial questions with generic follow-up prompts, such as 'please elaborate', 'any caveats?', or 'give me more detail', and score whether they elicit the same bias acknowledgements, unanswerability admissions, and counterevidence; if the generic prompts do, the metacognitive wording is not the causal ingredient. Also measure whether the model's answer to the original question changes after the follow-up, which the paper does not report.

Watch

Extended reading notes

Core claim

On its own terms, the discovery is that an instruction-tuned LLM holds latent knowledge about the limitations of its own outputs, and that a single metacognitive follow-up question can make that knowledge explicit. Given the answer to a word-association prompt, the model acknowledges that its pairings are stereotypically biased; given a multiple-choice medical question about a fictional organ, it recognizes that the premise is fictitious and explains why the question cannot be answered; given a summary of the too-much-choice effect, it volunteers that meta-analyses found the effect size near zero. Because the model's autoregressive architecture can only evaluate content that has already been produced, the paper argues, making the initial output explicit and then inviting critique is what allows the latent knowledge to be used. The prompt thereby converts implicit bias into explicit output the user can see and act on, and it reveals cases where the model's interpretation of a user's question differs from the model's own later account of what it knew.

Load-bearing premise

The paper assumes that the self-critique triggered by the exact wording 'could you be wrong?' is doing the work, and that making a bias explicit counts as debiasing, rather than comparing against generic follow-up prompts or testing whether the original answer actually changes.

Editorial extensions

If this is right

Users can expose implicit bias in LLM word associations without specialized bias tests, by following any answer with the prompt 'could you be wrong?'.
A poor score on 'I don't know' multiple-choice questions does not prove an LLM lacks metacognitive knowledge; the capacity appears when self-critique is invited.
Factual summaries that omit failed replications or null meta-analytic results can be corrected on demand, giving users access to counter-evidence before relying on the summary.
Iterating the prompt can surface less prominent biases, such as omission bias in moral decisions, that the initial answer and even a first critique miss.
Because the intervention is a language prompt rather than a model-specific adjustment, the same strategy has a chance of transferring to future models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the paper does not compare 'could you be wrong?' with generic elaboration prompts such as 'please explain further' or 'list caveats', so whether the metacognitive wording itself is causal, rather than any request for more text, remains untested.
Editorial inference: because the demonstrations record only the follow-up text, not a revised answer to the original question, a quantitative test of debiasing would compare initial and final answers and measure whether accuracy or fairness improves.
Editorial inference: the strategy's success presumably depends on the model's training data containing critique and counter-evidence; a model trained on filtered or narrower data would have less latent knowledge to surface.
Editorial inference: the same prompt could be embedded in a user-interface tool that automatically follows any high-stakes LLM answer with an adversarial check, but its effect on human decisions should be tested directly rather than assumed from the generated text.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick read on arXiv:2507.10124. The prompt 'could you be wrong?' does elicit genuinely useful self-critical text from ChatGPT-4o in the three demonstrated cases: biased word associations, a fictional-disease medical question, and the too-much-choice effect. The transcripts are verbatim, the tasks are drawn from recent real papers, and the connection to the human dialectical bootstrapping literature is legitimate and well sourced. That is the actual contribution, and it is real but narrow.

What the paper does not do is show debiasing. The initial answers are never revised; the critique is appended. There is no measure of whether the final output or downstream human decision improves. The title and discussion call it 'a practical and effective tool for debiasing... LLMs,' which overreaches the evidence. The abstract is more carefully worded; the discussion is not. The stress-test note is right on this.

Soft spots, in order of severity. First, no baseline comparison. A generic follow-up like 'please elaborate' or 'any caveats?' might produce similar self-critique, and nothing here shows 'could you be wrong?' is special. Second, the repeated-runs claim (>10 per case) is unverifiable: no logs, no counts, no counter-examples. Third, the paper does not engage the existing LLM self-refinement literature—Self-Refine, reflexion, self-consistency—so the novelty is oversold. Fourth, the human literature's dialectical bootstrapping aggregates two judgments because the second judgment improves accuracy; here there is no second judgment and no aggregation, so the psychology bridge is suggestive, not demonstrative.

None of this makes the paper worthless. As a short demonstration it is honest about what is shown—'produces additional information'—and the cases are well chosen. The right outcome: send to peer review as a short report, requiring the authors to either correct the framing to 'eliciting self-critical commentary' or add a small quantitative evaluation, such as one revised-answer condition or a baseline prompt comparison. I would not desk-reject; I would referee it with expectation of major revision or re-scope.

Referee Report

3 major / 6 minor

Summary. The paper argues that a metacognitive follow-up prompt from the human decision-making literature, 'could you be wrong?', can debias large language models by eliciting self-critical information. Drawing on human debiasing strategies (e.g., dialectical bootstrapping and premortems), the author applies the prompt after ChatGPT-4o's responses to three tasks: a word-association test of implicit bias from Bai et al. (2025), a medical multiple-choice question about a fictional organ from Griot et al. (2025), and a request to summarize the 'too much choice' effect. The three verbatim transcripts show that the model produces additional text identifying biases, counter-evidence, and alternatives. The paper concludes that the prompt is 'a practical and effective tool for debiasing both humans and, as demonstrated here, LLMs,' without quantifying bias reduction, comparing against control prompts, or testing downstream decisions.

Significance. The proposed transfer of metacognitive prompting from human judgment research to LLMs is original and potentially useful, and the paper is commendably transparent: it specifies the model, date, and settings, reports multiple runs, and provides verbatim excerpts, including a sanity check with a novel fictional disease. The phenomenon demonstrated—that follow-up questioning can make LLMs articulate limitations of their own answers—is real and worth reporting. However, the significance as a debiasing result is currently unestablished: the transcripts show appended self-critique, not reduced bias in outputs or improved human decisions. The paper is a proof-of-concept for prompt-elicited self-criticism, not evidence for the debiasing claim made in the title, abstract, and Discussion.

major comments (3)

[Discussion; Figures 1–3] The Discussion's claim that 'could you be wrong' is 'a practical and effective tool for debiasing both humans and, as demonstrated here, LLMs' is not supported by the data in Figures 1–3, because none of the demonstrations measures a reduction in bias. In each figure the model's initial answer is left untouched, and the follow-up prompt produces only additional self-critical text; the paper never asks the model to revise its answer or compares pre- and post-critique outputs on any bias metric. Debiasing requires showing that the biased behavior changes, so the experiments as presented establish only that LLMs can generate metacognitive commentary when prompted. I recommend either reframing the core claim to 'elicits self-critical information' or adding a revision step and a quantitative bias measurement.
[Discriminatory bias: Bai et al., 2025; Figures 1–3] The causal role of the metacognitive phrasing is untested because the paper lacks any control prompt. The follow-up always uses 'could you be wrong?', so the additional text could be elicited by any generic request for elaboration, such as 'please elaborate' or 'what are some caveats?'. In addition, the manuscript states that each case was run 'multiple times (>10)' with 'qualitatively consistent responses' across four models, but no data or negative examples are provided, and the author shows only a single transcript per case. Adding a control condition and a systematic summary of all runs, including any failures, is necessary to support the attribution of the effect to metacognitive prompting and to rule out selective reporting.
[Metacognitive bias: Griot et al. 2025; Evidence omission: The too much choice effect] The claimed improvements in metacognition and factual accuracy are not quantified. In the Glianorex/Clapzym Morphistic Disorder example, the model still selects option A even after acknowledging that the condition is fictional, so its behavior on the benchmark's central criterion is unchanged; in the too-much-choice example, ChatGPT does not correct its initial summary but only adds caveats after prompting. The paper reports no error rates, no calibration scores, no revised-answer comparison, and no human-subject evaluation of decision quality, so the abstract's promise of 'improving human decision making' is unmeasured.

minor comments (6)

[Introduction, final paragraph before 'Discriminatory bias'] The word 'disciminatory' appears in the sentence 'I apply this prompt following a set of prompts that have been used to reveal disciminatory bias'; it should be 'discriminatory'.
[Metacognitive bias: Griot et al. 2025] In the paragraph beginning 'The author's argue that not choosing...', the correct plural is 'The authors argue'; the current phrasing contains a typographical error.
[Evidence omission: The too much choice effect] In the first paragraph, 'enahnce' should be 'enhance' in 'how a metacognitive prompt can enahnce an LLM's capacity'.
[Throughout] The manuscript uses 'LLM's' as a plural in several places (e.g., 'LLM's often respond to prompts with stereotypical and popular responses'); the correct plural is 'LLMs'.
[Figure 2] The text notes that only the beginning of the model's response is shown, but the figure caption implies the full chat thread is displayed; consider adding the complete response as supplementary material for verification.
[Metacognitive bias: Griot et al. 2025] The claim that the novel fictional disease check rules out training-data leakage is too strong; a more convincing test would include several held-out fictional terms and repeated generations.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the prompt demonstration is empirical and not derived from its own conclusion.

full rationale

The paper contains no fitted parameters, no equations, and no derivation chain in which an output quantity is constructed from the input quantity. The central demonstration is empirical: verbatim transcripts show that prompting ChatGPT-4o with 'could you be wrong?' after an initial response elicits additional self-critical text. The claimed mechanism (LLMs contain latent knowledge about their biases that can be surfaced by a metacognitive prompt) is a psychological hypothesis, not a tautology built into the prompt's definition. The cited sources for the bias tasks (Bai et al., Griot et al.) and for the failure of the too-much-choice effect (Scheibehenne et al.) are external benchmarks, not the author's own prior work, so no self-citation chain is load-bearing. Although the Discussion's phrase 'effective tool for debiasing' goes beyond what the transcripts show, and although the lack of a control prompt such as 'please elaborate' or any before/after answer correction is a real methodological limitation, that is a validity and generalizability gap, not circularity. The paper's abstract itself carefully limits the claim to producing additional information, and therefore the demonstration does not reduce by construction to its inputs.

Assumptions & free parameters 0 free parameters · 2 assumptions · 0 invented entities

No free parameters and no invented entities. The paper's assumptions are about how LLMs process prompts (domain assumption) and about the adequacy of anecdotal evidence (ad hoc).

assumptions (2)

domain assumption LLMs only evaluate knowledge once it is made explicit in their output; latent knowledge is otherwise inaccessible to the model's own reasoning.
Introduced in the Introduction ('LLMs are not self-aware... biases must be addressed during training or managed at the prompt level') and used to justify the need for a follow-up prompt.
ad hoc to paper Qualitative consistency across '>10' runs and four models, reported without underlying data, is treated as sufficient evidence for the prompt's generality.
Stated on page 4 ('each of the cases... was run multiple times (>10) and in every case the prompt led to qualitatively consistent responses') but no data are shown.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Could you be wrong: Debiasing LLMs using a metacognitive prompt for improving human decision making." pith.science (2026). https://pith.science/paper/N26QIUDJ

@misc{pith2026250710124,
  author       = {Pith},
  title        = {Pith review of: Could you be wrong: Debiasing LLMs using a metacognitive prompt for improving human decision making},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N26QIUDJ}},
  note         = {Machine review of arXiv:2507.10124}
}

read the original abstract

Identifying bias in LLMs is ongoing. Because they are still in development, what is true today may be false tomorrow. We therefore need general strategies for debiasing that will outlive current models. Strategies developed for debiasing human decision making offer one promising approach as they incorporate an LLM-style prompt intervention designed to bring latent knowledge into awareness during decision making. LLMs trained on vast amounts of information contain information about potential biases, counter-arguments, and contradictory evidence, but that information may only be brought to bear if prompted. Metacognitive prompts developed in the human decision making literature are designed to achieve this, and as I demonstrate here, they show promise with LLMs. The prompt I focus on here is "could you be wrong?" Following an LLM response, this prompt leads LLMs to produce additional information, including why they answered as they did, errors, biases, contradictory evidence, and alternatives, none of which were apparent in their initial response. Indeed, this metaknowledge often reveals that how LLMs and users interpret prompts are not aligned. Here I demonstrate this prompt using a set of questions taken from recent articles about LLM biases, including implicit discriminatory biases and failures of metacognition. "Could you be wrong" prompts the LLM to identify its own biases and produce cogent metacognitive reflection. I also present another example involving convincing but incomplete information, which is readily corrected by the metacognitive prompt. In sum, this work argues that human psychology offers a new avenue for prompt engineering, leveraging a long history of effective prompt-based improvements to human decision making.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

35 extracted references · 32 canonical work pages

[1]

G., Ross, L

Lord, C. G., Ross, L. & Lepper, M. R. Biased assimilation and attitude polariza- tion: The effects of prior theories on subsequently considered evidence.Journal of Personality and Social Psychology 37, 2098 (1979)

work page 1979
[2]

& Kahneman, D

Tversky, A. & Kahneman, D. Availability: A heuristic for judging frequency and probability. Cognitive Psychology 5, 207–232 (1973)

work page 1973
[3]

Thinking, Fast and Slow

Kahneman, D. Thinking, Fast and Slow . (Macmillan, 2011)

work page 2011
[4]

Rich, A. S. & Gureckis, T. M. Lessons for artificial intelligence from the study of natural stupidity. Nature Machine Intelligence 1, 174–180 (2019)

work page 2019
[5]

E., Hertwig, R

Gigerenzer, G. E., Hertwig, R. E. & Pachur, T. E.Heuristics: The Foundations of Adaptive Behavior. (Oxford University Press, 2011)

work page 2011
[6]

Larrick, R. P. Debiasing. Blackwell Handbook of Judgment and Decision Making 316–338 (2004)

work page 2004
[7]

& Lieder, F

Cheung, V., Maier, M. & Lieder, F. Large language models show amplified cognitive biases in moral decision-making.Proceedings of the National Academy of Sciences 122, e2412015122 (2025)

work page 2025
[8]

Caliskan, A., Bryson, J. J. & Narayanan, A. Semantics derived automatically from language corpora contain human-like biases.Science 356, 183–186 (2017)

work page 2017

Show all 35 references

[9]

& Yuksel, D

Griot, M., Hemptinne, C., Vanderdonckt, J. & Yuksel, D. Large language models lack essential metacognition for reliable medical reasoning.Nature Communications 16, 642 (2025)

2025
[10]

T., Yao, S., Friedman, D., Hardy, M

McCoy, R. T., Yao, S., Friedman, D., Hardy, M. D. & Griffiths, T. L. Embers of autoregression show how large language models are shaped by the problem they are trained to solve.Proceedings of the National Academy of Sciences 121, e2322420121 (2024)

2024
[11]

& Griffiths, T

Bai, X., Wang, A., Sucholutsky, I. & Griffiths, T. L. Explicitly unbiased large language models still form biased associations.Proceedings of the National Academy of Sciences 122, e2416228122 (2025)

2025
[12]

& Rettenberger, L

Exler, D., Schutera, M., Reischl, M. & Rettenberger, L. Large means left: Political bias in large language models increases with their number of parameters.arXiv preprint arXiv:2505.04393 (2025)

2025 arXiv
[13]

Vaswani, A. et al. Attention is all you need. Advances in Neural Information Processing Systems 30, (2017)

2017
[14]

& Mazibuko, S

Durrheim, K., Schuld, M., Mafunda, M. & Mazibuko, S. Using word embeddings to investigate cultural biases.British Journal of Social Psychology 62, 617–629 (2023)

2023
[15]

Christiano, P. F.et al. Deep reinforcement learning from human preferences.Advances in Neural Information Processing Systems 30, (2017). COULD YOU BE WRONG 11

2017
[16]

Bai, Y. et al. Constitutional ai: Harmlessness from ai feedback. arXiv preprint arXiv:2212.08073 (2022)

2022 arXiv
[17]

Arkes, H. R. Costs and benefits of judgment errors: Implications for debiasing. Psychological Bulletin 110, 486 (1991)

1991
[18]

B., Milkman, K

Soll, J. B., Milkman, K. L. & Payne, J. W. A user’s guide to debiasing.The Wiley Blackwell handbook of judgment and decision making 2, 924–951 (2015)

2015
[19]

& Choshen-Hillel, S

Yaniv, I. & Choshen-Hillel, S. When guessing what another person would say is better than giving your own opinion: Using perspective-taking to improve advice-taking. Journal of Experimental Social Psychology 48, 1022–1028 (2012)

2012
[20]

A.,et al

Newell, A., Simon, H. A.,et al. Human Problem Solving . vol. 104 (Prentice-hall Englewood Cliffs, NJ, 1972)

1972
[21]

F., Nijstad, B

Rietzschel, E. F., Nijstad, B. A. & Stroebe, W. Relative accessibility of domain knowledge and creativity: The effects of knowledge activation on the quantity and originality of generated ideas.Journal of experimental social psychology 43, 933–946 (2007)

2007
[22]

S., Reid, M., Matsuo, Y

Kojima, T., Gu, S. S., Reid, M., Matsuo, Y. & Iwasawa, Y. Large language models are zero-shot reasoners. Advances in Neural Information Processing Systems 35, 22199–22213 (2022)

2022
[23]

Large language model guided tree-of-thought

Long, J. Large language model guided tree-of-thought. arXiv preprint arXiv:2305.08291 (2023)

2023 arXiv
[24]

G., Lepper, M

Lord, C. G., Lepper, M. R. & Preston, E. Considering the opposite: A corrective strategy for social judgment.Journal of Personality and Social Psychology 47, 1231 (1984)

1984
[25]

Hirt, E. R. & Markman, K. D. Multiple explanation: A consider-an-alternative strategy for debiasing judgments.Journal of Personality and Social Psychology 69, 1069 (1995)

1995
[26]

& Fischhoff, B

Koriat, A., Lichtenstein, S. & Fischhoff, B. Reasons for confidence. Journal of Experimental Psychology: Human learning and memory 6, 107 (1980)

1980
[27]

Herzog, S. M. & Hertwig, R. The wisdom of many in one mind: Improving individual judgments with dialectical bootstrapping.Psychological Science 20, 231–237 (2009)

2009
[28]

J., Edward Russo, J

Mitchell, D. J., Edward Russo, J. & Pennington, N. Back to the future: Temporal perspective in the explanation of events.Journal of Behavioral Decision Making 2, 25–38 (1989)

1989
[29]

Performing a project premortem

Klein, G. Performing a project premortem. Harvard Business Review 85, 18–19 (2007)

2007
[30]

& Todd, P

Scheibehenne, B., Greifeneder, R. & Todd, P. M. Can there ever be too many options? A meta-analytic review of choice overload.Journal of Consumer Research 37, 409–425 (2010)

2010
[31]

S., Bastos, R

Schoenegger, P., Tuminauskaite, I., Park, P. S., Bastos, R. V. S. & Tetlock, P. E. Wisdom of the silicon crowd: LLM ensemble prediction capabilities rival human crowd accuracy. Science Advances 10, eadp1528 (2024). COULD YOU BE WRONG 12

2024
[32]

& Cambria, E

Liu, L., Zhang, D., Li, S., Zhou, G. & Cambria, E. Two heads are better than one: Zero-shot cognitive reasoning via multi-LLM knowledge fusion. inProceedings of the 33rd ACM international conference on information and knowledge management 1462–1472 (2024)

2024
[33]

Why I write

Didion, J. Why I write. New York Times Book Review 5, 98–99 (1976)

1976
[34]

& Byrne, M

Fenn, K. & Byrne, M. The key principles of cognitive behavioural therapy.InnovAiT 6, 579–585 (2013)

2013
[35]

& Todd, P

Hertwig, R. & Todd, P. M. More is not always better: The benefits of cognitive limits. Thinking: Psychological perspectives on reasoning, judgment and decision making 213–231 (2003)

2003

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

G., Ross, L

Lord, C. G., Ross, L. & Lepper, M. R. Biased assimilation and attitude polariza- tion: The effects of prior theories on subsequently considered evidence.Journal of Personality and Social Psychology 37, 2098 (1979)

work page 1979

[2] [2]

& Kahneman, D

Tversky, A. & Kahneman, D. Availability: A heuristic for judging frequency and probability. Cognitive Psychology 5, 207–232 (1973)

work page 1973

[3] [3]

Thinking, Fast and Slow

Kahneman, D. Thinking, Fast and Slow . (Macmillan, 2011)

work page 2011

[4] [4]

Rich, A. S. & Gureckis, T. M. Lessons for artificial intelligence from the study of natural stupidity. Nature Machine Intelligence 1, 174–180 (2019)

work page 2019

[5] [5]

E., Hertwig, R

Gigerenzer, G. E., Hertwig, R. E. & Pachur, T. E.Heuristics: The Foundations of Adaptive Behavior. (Oxford University Press, 2011)

work page 2011

[6] [6]

Larrick, R. P. Debiasing. Blackwell Handbook of Judgment and Decision Making 316–338 (2004)

work page 2004

[7] [7]

& Lieder, F

Cheung, V., Maier, M. & Lieder, F. Large language models show amplified cognitive biases in moral decision-making.Proceedings of the National Academy of Sciences 122, e2412015122 (2025)

work page 2025

[8] [8]

Caliskan, A., Bryson, J. J. & Narayanan, A. Semantics derived automatically from language corpora contain human-like biases.Science 356, 183–186 (2017)

work page 2017