REVIEW 3 major objections 5 minor 19 references
A Cross-Domain Study of the Use of Persuasion Techniques in Online Disinformation
T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper shows that disinformation campaigns tailor their persuasion techniques to the topic, with differences large enough to be measured and exploited.
desk verdict A genuinely new cross-domain persuasion-technique comparison whose headline odds ratios rest on an unvalidated classifier; the paper deserves a serious referee but should be read as exploratory until the measurement is checked. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying machinery is a supervised multi-label classifier for sixteen persuasion techniques, built by fine-tuning a large transformer language model on a multilingual news-article benchmark with all languages translated to English. The classifier labels every sentence in four disinformation corpora; the paper then computes odds ratios for each technique in one domain versus its average presence in the other three and keeps only results significant by Fisher's exact test at p<0.05. For the adaptation question, it correlates technique presence with categories from a linguistic and psychological word-count lexicon, focusing on the four techniques most over-represented in climate-change disinformation. The named persuasion techniques (Appeal to Authority, Appeal to Hypocrisy, and the rest) are the central objects being measured.
What would settle it
Manually annotate a random sample of sentences from each of the four datasets for the sixteen techniques, compare the classifier's per-domain error rates, and recompute the odds ratios on the human labels; if the domain-specific gaps shrink or vanish, the reported differences are artifacts of uneven misclassification.
Extended reading notes
Core claim
The paper's core discovery is that persuasion-technique use in disinformation is domain-dependent rather than uniform. Among statistically significant odds ratios, Repetition is about ten times more likely in Islamic-issues disinformation than elsewhere, Appeal to Authority is about eleven times more likely in climate-change disinformation, and Appeal to Hypocrisy is about nineteen times more likely in Russo-Ukrainian war disinformation, with Questioning the Reputation, Guilt by Association, and Flag Waving also concentrated there. Beyond frequencies, the climate-change case study shows the same technique can be adapted in form: Appeal to Authority there is packaged in longer, more analytic, formal sentences, while Exaggeration-Minimisation leans on certitude words and moral and cultural references. The conclusion is that disinformation is tailored to audience and theme, not just in what it says but in how it persuades.
Load-bearing premise
The load-bearing premise is that the classifier's labels on the four target datasets are accurate enough that its per-domain differences are real, because the classifier was trained on a different news corpus and is never validated on those datasets.
Editorial extensions
If this is right
- Detection models should be trained or calibrated per domain, because a technique that is rare overall can be a strong signal inside one topic, such as Appeal to Hypocrisy in war disinformation.
- Fact-checkers can prepare rebuttal templates that name the specific persuasion move dominant in a domain, such as deflating false appeals to authority in climate denial.
- Media-literacy programs can teach that the same underlying technique changes its surface form across topics, so recognition must focus on function rather than wording.
- Because Doubt and Loaded Language appear across all four domains, any general countermeasure should start with these two as baseline manipulation strategies.
Reading between the lines
- A natural, untested next step is to run the same odds-ratio pipeline on other high-stakes domains such as elections, migration, or health to see whether every domain develops its own signature technique; the paper's method makes this a direct replication.
- Because the classifier was built and tested on news-style articles, applying it to short social-media posts could mislabel persuasion; verifying the odds ratios on platform-native text would test how far the result generalizes.
- One could check whether the climate-change word-count correlates reflect strategic adaptation or simply topic vocabulary by comparing disinformation to non-disinformative climate text matched for topic; the paper does not make that comparison.
- The Russo-Ukrainian war odds ratios come from machine-translated multilingual articles, so a natural test is whether the same signature techniques appear in the original-language texts before translation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents a cross-domain quantitative study of persuasion techniques in disinformation. Four datasets (CIDII, COVID-19, Climate Fever, EUvsDisinfo) are annotated at the sentence level with 16 persuasion techniques using the Razuvayevskaya et al. classifier, which was trained on SemEval-2023 news articles. The authors compute proportions and odds ratios across domains to answer RQ1, reporting large disparities (e.g., OR 19.28 for Appeal to Hypocrisy in Russo-Ukrainian war disinformation, OR 11.37 for Appeal to Authority in climate change). They then conduct a case study for RQ2 on climate change, correlating LIWC features with four domain-specific techniques to argue for contextual adaptation. The paper concludes that persuasion techniques are used disproportionately and adapted linguistically and psychologically across disinformation domains.
Significance. If the reported cross-domain differences are real, the study would be a useful contribution to the computational propaganda literature, providing a large-scale, publicly reproducible comparison that goes beyond single-domain analyses. The authors make their code and supplementary material openly available and apply a state-of-the-art classifier, which are strengths. However, the significance is conditional on the measurement instrument being valid on the target corpora; the current manuscript does not demonstrate that, so the empirical claims are not yet established. The topic is timely and the datasets are diverse, which adds potential value if the methodological gaps are addressed.
major comments (3)
- [§2.1, §2.2] The persuasion classifier is trained on SemEval-2023 news articles in English translation and then applied without any validation to the four target corpora, which differ in genre, topic, and, for EUvsDisinfo, machine-translated text. The central claim of the paper is differential (odds ratios between domains), and such differences can be produced by domain-dependent misclassification even when the classifier has high overall accuracy on its original benchmark. The authors should provide a manual gold-standard sample (e.g., 200–300 sentences per domain) with per-domain precision, recall, and a confusion matrix for the 16 techniques, and then re-examine whether the reported odds ratios in Table 2 survive after correcting for classifier bias. Without this, the odds ratios of 19.28 and 11.37 could be artifacts of systematic mislabeling in specific domains.
- [Table 2, §3] The statistical analysis treats sentences as independent observations in Fisher's exact tests, but sentences are nested within articles/documents, and persuasion techniques are likely correlated within the same article. This clustering can produce artificially small p-values and overly narrow conclusions. Additionally, the odds ratio is computed by comparing the odds in one domain to the 'average proportion' in the other three domains, which is not a standard odds ratio; the denominator should be the odds, not the proportion, and the construction should be defined explicitly. The authors should use a method that accounts for clustering (e.g., mixed-effects logistic regression with article as a random effect, or cluster-bootstrap by article) and report confidence intervals for the odds ratios. Without this, the statistical significance claims are not reliable.
- [§2.2, Table 1] The four datasets are described as disinformation domains, but they differ in text type: CIDII appears to contain social media posts, COVID-19 and EUvsDisinfo contain news articles (with EUvsDisinfo in multiple languages), and Climate Fever contains fact-checked claims (which may be shorter, extracted claims rather than full articles). These genre differences are a confound for the cross-domain comparison: observed differences in persuasion techniques may reflect the text genre (e.g., a 150-word translated news article vs. a short social media post) rather than the thematic domain. The authors should acknowledge this limitation and, ideally, restrict analyses to comparable text types or control for text length and genre in the statistical model.
minor comments (5)
- [Table 1] The table header contains a typo: 'Russo-Ukranian War' should be 'Russo-Ukrainian War'.
- [§2.1] The claim that the classifier is 'current ranked first on the post-competion leaderboard' should include a citation or a direct URL to the leaderboard; 'post-competion' is also a typo for 'post-competition'.
- [§2.2] The use of GPT-4o for translating EUvsDisinfo texts is mentioned but not validated; please report any quality checks or at least acknowledge that translation errors could affect classifier performance.
- [§3.2, Figure 2] The LIWC correlation analysis involves a large number of correlations, and the paper states that asterisks mark p<0.05 coefficients. A multiple-comparison correction (e.g., Bonferroni or FDR) is needed to support the claim of statistical significance.
- [General] The paper uses a placeholder ACM reference format with 'Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009' in the References section; this should be removed or replaced with the actual submission information.
Circularity Check
No circularity: the cross-domain odds ratios are computed by applying a pre-trained, externally benchmarked classifier to external datasets; the main threats are validity, not circularity.
full rationale
The paper's derivation chain is not circular. The central quantities (odds ratios for 16 persuasion techniques across four domains) are obtained by applying the Razuvayevskaya et al. classifier, trained on SemEval-2023 data, to four external datasets (CIDII, COVID-19, Climate Fever, EUvsDisinfo). No parameter is fitted to the target datasets, and the persuasion technique labels are not defined in terms of the reported odds ratios; the odds ratios are computed directly from the classifier's fixed outputs using Fisher's exact test. The classifier itself is justified by an external benchmark (ranked first on the SemEval-2023 post-competition leaderboard), so the self-citation to Ref. [17] is supported by independent evidence. The EUvsDisinfo dataset (Ref. [13]) is also a co-authored resource, but it is used as raw input text, not as a fitted target. The LIWC correlation case study (RQ2) is a post-hoc descriptive analysis of the same predicted labels; while its interpretability depends on the classifier's validity on these domains, it does not reduce to the paper's inputs by construction. The main weakness—that the classifier is never validated on a manual gold sample from the four target corpora—is a domain-shift/validity concern that could undermine the conclusions if classifier errors are domain-dependent, but it is not a circularity in the technical sense defined here. No specific equation or definition makes the reported results true by construction.
Assumptions & free parameters
assumptions (5)
- domain assumption The persuasion classifier's output is treated as ground truth on the four target domains.
- domain assumption Sentences are independent units for Fisher's exact test.
- domain assumption GPT-4o translations preserve persuasion techniques.
- domain assumption The four datasets are representative of their domains.
- domain assumption The SemEval-2023 persuasion taxonomy is applicable to all four domains.
Cite this review
Pith. "Pith review of A Cross-Domain Study of the Use of Persuasion Techniques in Online Disinformation." pith.science (2026). https://pith.science/paper/HGPSIG67
@misc{pith2026241215098,
author = {Pith},
title = {Pith review of: A Cross-Domain Study of the Use of Persuasion Techniques in Online Disinformation},
year = {2026},
howpublished = {\url{https://pith.science/paper/HGPSIG67}},
note = {Machine review of arXiv:2412.15098}
}
read the original abstract
Disinformation, irrespective of domain or language, aims to deceive or manipulate public opinion, typically through employing advanced persuasion techniques. Qualitative and quantitative research on the weaponisation of persuasion techniques in disinformation has been mostly topic-specific (e.g., COVID-19) with limited cross-domain studies, resulting in a lack of comprehensive understanding of these strategies. This study employs a state-of-the-art persuasion technique classifier to conduct a large-scale, multi-domain analysis of the role of 16 persuasion techniques in disinformation narratives. It shows how different persuasion techniques are employed disproportionately in different disinformation domains. We also include a detailed case study on climate change disinformation, highlighting how linguistic, psychological, and cultural factors shape the adaptation of persuasion strategies to fit unique thematic contexts.
Figures
Reference graph
Works this paper leans on
-
[1]
Zainab Alimoradi et al. 2022. Fear of COVID-19 and its association with mental health-related factors. (March 2022). https://doi.org/10.1192/bjo.2022.26
-
[2]
Samy Amanatullah. 2023. Tell Us How You Really Feel: Analyzing Pro- Kremlin Propaganda Devices & Narratives to Identify Sentiment Implications. (2023). https://www.illiberalism.org/tell-us-how-you-really-feel-analyzing-pro- kremlin-propaganda-devices-narratives-to-identify-sentiment-implications/
work page 2023
-
[3]
Ryan L Boyd et al. 2022. The development and psychometric properties of LIWC-
work page 2022
-
[4]
John Cook. 2019. Understanding and Countering Misinformation About Climate Change. https://doi.org/10.4018/978-1-5225-8535-0.ch016
-
[5]
Giovanni Da San Martino et al. 2020. A Survey on Computational Propaganda Detection. https://doi.org/10.24963/ijcai.2020/672
-
[6]
Thomas Diggelmann et al. 2021. CLIMATE-FEVER: A Dataset for Verification of Real-World Climate Claims. https://doi.org/10.48550/arXiv.2012.00614
-
[7]
Lisa K. Fazio et al. 2015. Knowledge does not protect against illusory truth. (Oct. 2015). https://doi.org/10.1037/xge0000098
-
[8]
Gloria Gennaro and Elliott Ash. 2022. Emotion and Reason in Political Language. 132, 643 (April 2022). https://doi.org/10.1093/ej/ueab104
Show all 19 references
-
[9]
Suhaib Kh Hamed, Mohd Juzaiddin Ab Aziz, and Mohd Ridzwan Yaakub. 2023. Disinformation Detection About Islamic Issues on Social Media Using Deep Learning Techniques. (July 2023). https://doi.org/10.22452/mjcs.vol36no3.3
2023 doi
-
[10]
Lynn Hasher, David Goldstein, and Thomas Toppino. 1977. Frequency and the conference of referential validity. (Feb. 1977). https://doi.org/10.1016/S0022- 5371(77)80012-1
1977 doi
-
[11]
Jowett and Victoria O’Donnell
Garth S. Jowett and Victoria O’Donnell. 2018. Propaganda & Persuasion. https: //books.google.com/books?id=vIC92PdJ0l0C
2018
-
[12]
Bonka Kotseva et al. 2023. Trend analysis of COVID-19 mis/disinformation narratives–A 3-year study. (Nov. 2023). https://doi.org/10.1371/journal.pone. 0291423
2023 doi
-
[13]
Leite et al
João A. Leite et al. 2024. EUvsDisinfo: A Dataset for Multilingual Detection of Pro-Kremlin Disinformation in News Articles. https://doi.org/10.1145/3627673. 3679167
2024 doi
-
[14]
Parth Patwa et al. 2021. Fighting an Infodemic: COVID-19 Fake News Dataset. https://doi.org/10.1007/978-3-030-73696-5_3
2021 doi
-
[15]
Jakub Piskorski et al. 2023. SemEval-2023 Task 3: Detecting the Category, the Framing, and the Persuasion Techniques in Online News in a Multi-lingual Setup. https://doi.org/10.18653/v1/2023.semeval-1.317
2023 doi
-
[16]
Robert N Proctor and Schiebinger Londa. 2008. Agnotology: The making and unmaking of ignorance. https://philarchive.org/archive/PROATM
2008
-
[17]
Olesya Razuvayevskaya et al. 2024. Comparison between Parameter-Efficient Techniques and Full Fine-Tuning. (May 2024). https://doi.org/10.1371/journal. pone.0301738
2024 doi
-
[18]
Ivan Srba et al. 2024. A Survey on Automatic Credibility Assessment of Textual Credibility Signals in the Era of Large Language Models. https://doi.org/10. 48550/arXiv.2410.21360 Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009
2024 doi
-
[22]
https://doi.org/10.13140/RG
Austin, TX: University of Texas at Austin (2022). https://doi.org/10.13140/RG. 2.2.23890.43205
2022
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.