Pith. sign in

REVIEW 2 cited by

Primacy Effect of ChatGPT

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.13206 v2 pith:CGOG2FMR submitted 2023-10-20 cs.CL cs.AI

Yiwei Wang , Yujun Cai , Muhao Chen , Yuxuan Liang , Bryan Hooi This is my paper · ORCID
classification cs.CLcs.AI
keywords chatgptlabelsanswerearliereffectlanguagelargepositions
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Instruction-tuned large language models (LLMs), such as ChatGPT, have led to promising zero-shot performance in discriminative natural language understanding (NLU) tasks. This involves querying the LLM using a prompt containing the question, and the candidate labels to choose from. The question-answering capabilities of ChatGPT arise from its pre-training on large amounts of human-written text, as well as its subsequent fine-tuning on human preferences, which motivates us to ask: Does ChatGPT also inherits humans' cognitive biases? In this paper, we study the primacy effect of ChatGPT: the tendency of selecting the labels at earlier positions as the answer. We have two main findings: i) ChatGPT's decision is sensitive to the order of labels in the prompt; ii) ChatGPT has a clearly higher chance to select the labels at earlier positions as the answer. We hope that our experiments and analyses provide additional insights into building more reliable ChatGPT-based solutions. We release the source code at https://github.com/wangywUST/PrimacyEffectGPT.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Fragile Preferences: A Deep Dive Into Order Effects in Large Language Models

    cs.AI 2025-06 unverdicted novelty 6.0 of 10

    LLMs show a quality-dependent position bias, favoring the first option for high-quality choices and later options for low-quality ones, and higher-temperature sampling can reveal the underlying preference.

  2. Who is in the Spotlight: The Hidden Bias Undermining Multimodal Retrieval-Augmented Generation

    cs.CL 2025-05 conditional novelty 4.0 of 10

    Multimodal RAG models show a U-shaped position bias in evidence ordering, and the bias grows logarithmically with the number of retrieved items, according to a new metric.

Pith tools