An LLM pre-annotation pipeline with span extraction and hierarchical labels improves human inter-annotator agreement and speed on Russian propaganda tweets, with smaller models distilled from the LLM outputs.
Are Large Language Models Good at Detecting Propaganda?
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Propagandists use rhetorical devices that rely on logical fallacies and emotional appeals to advance their agendas. Recognizing these techniques is key to making informed decisions. Recent advances in Natural Language Processing (NLP) have enabled the development of systems capable of detecting manipulative content. In this study, we look at several Large Language Models and their performance in detecting propaganda techniques in news articles. We compare the performance of these LLMs with transformer-based models. We find that, while GPT-4 demonstrates superior F1 scores (F1=0.16) compared to GPT-3.5 and Claude 3 Opus, it does not outperform a RoBERTa-CRF baseline (F1=0.67). Additionally, we find that all three LLMs outperform a MultiGranularity Network (MGN) baseline in detecting instances of one out of six propaganda techniques (name-calling), with GPT-3.5 and GPT-4 also outperforming the MGN baseline in detecting instances of appeal to fear and flag-waving.
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
Hybrid Annotation for Propaganda Detection: Integrating LLM Pre-Annotations with Human Intelligence
An LLM pre-annotation pipeline with span extraction and hierarchical labels improves human inter-annotator agreement and speed on Russian propaganda tweets, with smaller models distilled from the LLM outputs.