An LLM pre-annotation pipeline with span extraction and hierarchical labels improves human inter-annotator agreement and speed on Russian propaganda tweets, with smaller models distilled from the LLM outputs.
Bias in LLMs as Annotators: The Effect of Party Cues on Labelling Decision by Large Language Models
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Human coders are biased. We test similar biases in Large Language Models (LLMs) as annotators. By replicating an experiment run by Ennser-Jedenastik and Meyer (2018), we find evidence that LLMs use political information, and specifically party cues, to judge political statements. Not only do LLMs use relevant information to contextualize whether a statement is positive, negative, or neutral based on the party cue, they also reflect the biases of the human-generated data upon which they have been trained. We also find that unlike humans, who are only biased when faced with statements from extreme parties, LLMs exhibit significant bias even when prompted with statements from center-left and center-right parties. The implications of our findings are discussed in the conclusion.
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Hybrid Annotation for Propaganda Detection: Integrating LLM Pre-Annotations with Human Intelligence
An LLM pre-annotation pipeline with span extraction and hierarchical labels improves human inter-annotator agreement and speed on Russian propaganda tweets, with smaller models distilled from the LLM outputs.