{"id":"3712a378-a4f1-40e3-a36b-7e97ef5dc9e7","arxiv_id":"2412.15098","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":0,"one_line_summary":"Persuasion techniques are used disproportionately across disinformation domains, with climate change uniquely favoring Appeal to Authority and Exaggeration-Minimisation.","lead":"This study applies a state-of-the-art persuasion technique classifier to four disinformation datasets and finds that different domains rely on different persuasion tactics. It also shows how climate change disinformation adapts the style of those tactics to its scientific and cultural context.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central cross-domain comparisons rest on an out-of-domain persuasion classifier that is never validated on the four target corpora; domain-dependent misclassification could generate the reported odds ratios.","rationale":"The reader's weakest_assumption identifies exactly the load-bearing point. The paper's methodological core is a single classifier applied to four unseen corpora; every quantitative result in Table 2 and the case study in §3.2 inherits its error behavior. In an ideal setting, uniform error rates across domains would cancel in odds ratios, but the domains differ in genre, topic, and, for EUvsDisinfo, in source language and translation pipeline. These are precisely the conditions under which fine-tuned classifiers develop systematic, corpus-specific biases. The paper does provide some independent support: the classifier is state of the art, code is public, and the qualitative examples are plausible. But none of that establishes measurement invariance, which is the load-bearing assumption for cross-domain comparisons. My recommended check is modest and directly actionable: a 400-sentence gold sample and per-domain error analysis. Until that is done, the paper is a useful exploratory study, but its headline claim should be treated as conditional. I therefore do not change the reader's verdict.","tokens_in":6500,"tokens_out":3770,"duration_ms":35249,"concrete_test":"Construct a gold evaluation set of 100 sentences per domain (400 total) from the four target datasets, stratified by article and sentence length. Have two annotators, using the SemEval-2023 annotation guidelines, label each sentence for the 16 techniques; compute inter-annotator agreement. Run the Razuvayevskaya classifier on this set and report per-domain precision, recall, F1, and a confusion matrix. Then re-estimate the Table 2 odds ratios in two ways: (a) after removing sentences where humans and classifier disagree on the technique in question, and (b) using misclassification-adjusted counts. If per-domain F1 varies materially, or the confusion matrix reveals systematic errors aligned with the claimed odds ratios (e.g., climate sentences misread as Appeal to Authority, EUvsDisinfo sentences misread as Appeal to Hypocrisy), the cross-domain claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (RQ1, §3.1) is that disinformation domains differ in their use of persuasion techniques, with odds ratios as extreme as 19.28 for Appeal to Hypocrisy in the Russo-Ukrainian war and 11.37 for Appeal to Authority in climate change. The only instrument for measuring techniques is the Razuvayevskaya et al. classifier, trained on SemEval-2023 news articles in English translation (§2.1). It is applied directly to CIDII, COVID-19, Climate Fever, and EUvsDisinfo, with non-English EUvsDisinfo text machine-translated via GPT-4o (§2.2). No manual gold sample from any target corpus is used to estimate precision, recall, or a confusion matrix on these domains. Since the claim is differential, the critical condition is not that the classifier is accurate on average but that its errors are domain-invariant. That is implausible: topic shifts (scientific register in climate text, translated propaganda in EUvsDisinfo, social-media-like posts in CIDII) are exactly the kind of variation that changes a fine-tuned RoBERTa model's error patterns. A domain-specific bias, e.g., systematically labelling sentences containing 'Nazi' or 'they said one thing and did another' as Appeal to Hypocrisy regardless of the actual argumentative function, would produce the reported odds ratios without any real cross-domain difference. The classifier's leaderboard rank and the public code support the tool's general quality, not its validity on these out-of-distribution inputs.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a cross-domain quantitative study of persuasion techniques in disinformation. Four datasets (CIDII, COVID-19, Climate Fever, EUvsDisinfo) are annotated at the sentence level with 16 persuasion techniques using the Razuvayevskaya et al. classifier, which was trained on SemEval-2023 news articles. The authors compute proportions and odds ratios across domains to answer RQ1, reporting large disparities (e.g., OR 19.28 for Appeal to Hypocrisy in Russo-Ukrainian war disinformation, OR 11.37 for Appeal to Authority in climate change). They then conduct a case study for RQ2 on climate change, correlating LIWC features with four domain-specific techniques to argue for contextual adaptation. The paper concludes that persuasion techniques are used disproportionately and adapted linguistically and psychologically across disinformation domains.","tokens_in":6908,"tokens_out":2849,"duration_ms":27302,"significance":"If the reported cross-domain differences are real, the study would be a useful contribution to the computational propaganda literature, providing a large-scale, publicly reproducible comparison that goes beyond single-domain analyses. The authors make their code and supplementary material openly available and apply a state-of-the-art classifier, which are strengths. However, the significance is conditional on the measurement instrument being valid on the target corpora; the current manuscript does not demonstrate that, so the empirical claims are not yet established. The topic is timely and the datasets are diverse, which adds potential value if the methodological gaps are addressed.","major_comments":[{"comment":"The persuasion classifier is trained on SemEval-2023 news articles in English translation and then applied without any validation to the four target corpora, which differ in genre, topic, and, for EUvsDisinfo, machine-translated text. The central claim of the paper is differential (odds ratios between domains), and such differences can be produced by domain-dependent misclassification even when the classifier has high overall accuracy on its original benchmark. The authors should provide a manual gold-standard sample (e.g., 200–300 sentences per domain) with per-domain precision, recall, and a confusion matrix for the 16 techniques, and then re-examine whether the reported odds ratios in Table 2 survive after correcting for classifier bias. Without this, the odds ratios of 19.28 and 11.37 could be artifacts of systematic mislabeling in specific domains.","section":"§2.1, §2.2"},{"comment":"The statistical analysis treats sentences as independent observations in Fisher's exact tests, but sentences are nested within articles/documents, and persuasion techniques are likely correlated within the same article. This clustering can produce artificially small p-values and overly narrow conclusions. Additionally, the odds ratio is computed by comparing the odds in one domain to the 'average proportion' in the other three domains, which is not a standard odds ratio; the denominator should be the odds, not the proportion, and the construction should be defined explicitly. The authors should use a method that accounts for clustering (e.g., mixed-effects logistic regression with article as a random effect, or cluster-bootstrap by article) and report confidence intervals for the odds ratios. Without this, the statistical significance claims are not reliable.","section":"Table 2, §3"},{"comment":"The four datasets are described as disinformation domains, but they differ in text type: CIDII appears to contain social media posts, COVID-19 and EUvsDisinfo contain news articles (with EUvsDisinfo in multiple languages), and Climate Fever contains fact-checked claims (which may be shorter, extracted claims rather than full articles). These genre differences are a confound for the cross-domain comparison: observed differences in persuasion techniques may reflect the text genre (e.g., a 150-word translated news article vs. a short social media post) rather than the thematic domain. The authors should acknowledge this limitation and, ideally, restrict analyses to comparable text types or control for text length and genre in the statistical model.","section":"§2.2, Table 1"}],"minor_comments":[{"comment":"The table header contains a typo: 'Russo-Ukranian War' should be 'Russo-Ukrainian War'.","section":"Table 1"},{"comment":"The claim that the classifier is 'current ranked first on the post-competion leaderboard' should include a citation or a direct URL to the leaderboard; 'post-competion' is also a typo for 'post-competition'.","section":"§2.1"},{"comment":"The use of GPT-4o for translating EUvsDisinfo texts is mentioned but not validated; please report any quality checks or at least acknowledge that translation errors could affect classifier performance.","section":"§2.2"},{"comment":"The LIWC correlation analysis involves a large number of correlations, and the paper states that asterisks mark p<0.05 coefficients. A multiple-comparison correction (e.g., Bonferroni or FDR) is needed to support the claim of statistical significance.","section":"§3.2, Figure 2"},{"comment":"The paper uses a placeholder ACM reference format with 'Received 20 February 2007; revised 12 March 2009; accepted 5 June 2009' in the References section; this should be removed or replaced with the actual submission information.","section":"General"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within scope for cs.CY and the topic is appropriate for the venue, but the load-bearing measurement validation issue needs to be addressed. I also note that three authors are co-authors of the classifier paper cited as the method; while this is not inherently problematic, the paper should be careful not to overstate the classifier's generalizability, and the validation step I request would also address concerns about self-assessment. The statistical issues, particularly the independence assumption, are standard in NLP/quantitative discourse analysis and should be fixable with a reasonable revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the cross-domain comparison is new and the paper is worth a serious referee, but the headline effect sizes are not yet trustworthy because the measuring instrument is never validated on the four target corpora.\n\nWhat's genuinely useful: a shared 16-technique taxonomy applied consistently to four disinformation domains (Islamic issues, COVID-19, climate, Russo-Ukrainian war) yields the first direct cross-domain odds-ratio table I know of. The results are vivid — Appeal to Hypocrisy at 19.28 in the war corpus, Appeal to Authority at 11.37 in climate — and the qualitative examples help ground them. Code and data are public. The LIWC case study is a reasonable way to approach RQ2, even if it is descriptive.\n\nThe soft spots are concentrated in the measurement. The classifier (Razuvayevskaya et al.) was trained on SemEval-2023 news articles and is applied without any re-validation to social-media posts, scientific-adjacent climate text, and GPT-4o-translated EUvsDisinfo. Domain-dependent misclassification is not a footnote concern; it could manufacture the reported odds ratios. There are also no confidence intervals, and Fisher's exact tests treat sentences as independent even though they are nested in articles. The LIWC correlations are many and uncorrected. I'd call RQ1 an interesting hypothesis, not a demonstrated result.\n\nThe three co-authors on the classifier paper is a mild self-citation issue, but the classifier is public and leaderboard-ranked, so that is not the main problem.\n\nRecommendation: send to review. A good referee should ask for a small annotated validation sample per domain, clustered standard errors, and a more cautious reading of the ORs. With those changes the paper becomes a solid reference for cross-domain persuasion research.","headline":"A genuinely new cross-domain persuasion-technique comparison whose headline odds ratios rest on an unvalidated classifier; the paper deserves a serious referee but should be read as exploratory until the measurement is checked.","tokens_in":7223,"tokens_out":2398,"would_cite":true,"duration_ms":20228,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper shows that disinformation campaigns tailor their persuasion techniques to the topic, with differences large enough to be measured and exploited.","keywords":["disinformation","persuasion techniques","domain adaptation","cross-domain analysis","propaganda detection","climate change disinformation","Russo-Ukrainian war disinformation","odds ratio analysis"],"falsifier":"Manually annotate a random sample of sentences from each of the four datasets for the sixteen techniques, compare the classifier's per-domain error rates, and recompute the odds ratios on the human labels; if the domain-specific gaps shrink or vanish, the reported differences are artifacts of uneven misclassification.","tokens_in":6341,"feed_emoji":"🎭","tokens_out":9939,"duration_ms":77482,"temperature":0.7,"pith_summary":"This paper asks whether online disinformation has a single persuasion playbook or different ones depending on the topic, and answers by comparing COVID-19, climate change, Islamic-issues, and Russo-Ukrainian war disinformation. Using a classifier for sixteen persuasion techniques, it measures each technique's over- or under-use in one domain relative to the others, and finds that some techniques (Doubt, Loaded Language) appear everywhere while others concentrate in specific topics. It also shows, in a climate-change case study, that a technique's linguistic and psychological packaging shifts to fit the topic. The practical payoff is that detectors, fact-checkers, and media-literacy programs could target the persuasion signatures of each domain instead of treating all disinformation as identical.","feed_headline":"War fake news leans on hypocrisy; climate denial leans on authority","feed_subtitle":"A 16-tactic scan of four fake-news domains shows each campaign tailors its rhetoric—a sharper target for detectors","key_machinery":"The carrying machinery is a supervised multi-label classifier for sixteen persuasion techniques, built by fine-tuning a large transformer language model on a multilingual news-article benchmark with all languages translated to English. The classifier labels every sentence in four disinformation corpora; the paper then computes odds ratios for each technique in one domain versus its average presence in the other three and keeps only results significant by Fisher's exact test at p<0.05. For the adaptation question, it correlates technique presence with categories from a linguistic and psychological word-count lexicon, focusing on the four techniques most over-represented in climate-change disinformation. The named persuasion techniques (Appeal to Authority, Appeal to Hypocrisy, and the rest) are the central objects being measured.","core_discovery":"The paper's core discovery is that persuasion-technique use in disinformation is domain-dependent rather than uniform. Among statistically significant odds ratios, Repetition is about ten times more likely in Islamic-issues disinformation than elsewhere, Appeal to Authority is about eleven times more likely in climate-change disinformation, and Appeal to Hypocrisy is about nineteen times more likely in Russo-Ukrainian war disinformation, with Questioning the Reputation, Guilt by Association, and Flag Waving also concentrated there. Beyond frequencies, the climate-change case study shows the same technique can be adapted in form: Appeal to Authority there is packaged in longer, more analytic, formal sentences, while Exaggeration-Minimisation leans on certitude words and moral and cultural references. The conclusion is that disinformation is tailored to audience and theme, not just in what it says but in how it persuades.","pith_inferences":["A natural, untested next step is to run the same odds-ratio pipeline on other high-stakes domains such as elections, migration, or health to see whether every domain develops its own signature technique; the paper's method makes this a direct replication.","Because the classifier was built and tested on news-style articles, applying it to short social-media posts could mislabel persuasion; verifying the odds ratios on platform-native text would test how far the result generalizes.","One could check whether the climate-change word-count correlates reflect strategic adaptation or simply topic vocabulary by comparing disinformation to non-disinformative climate text matched for topic; the paper does not make that comparison.","The Russo-Ukrainian war odds ratios come from machine-translated multilingual articles, so a natural test is whether the same signature techniques appear in the original-language texts before translation."],"forward_implications":["Detection models should be trained or calibrated per domain, because a technique that is rare overall can be a strong signal inside one topic, such as Appeal to Hypocrisy in war disinformation.","Fact-checkers can prepare rebuttal templates that name the specific persuasion move dominant in a domain, such as deflating false appeals to authority in climate denial.","Media-literacy programs can teach that the same underlying technique changes its surface form across topics, so recognition must focus on function rather than wording.","Because Doubt and Loaded Language appear across all four domains, any general countermeasure should start with these two as baseline manipulation strategies."],"supporting_citations":[{"why":"Supplies the persuasion classifier used to label every sentence in the four disinformation datasets.","marker":"[17]"},{"why":"Defines the persuasion-technique taxonomy and the multilingual training data the classifier is built on.","marker":"[15]"},{"why":"Provides the Russo-Ukrainian war disinformation corpus, translated to English and filtered for war topics.","marker":"[13]"},{"why":"Provides the climate-change disinformation texts used in the frequency and case-study analyses.","marker":"[6]"},{"why":"Provides the Islamic-issues disinformation dataset.","marker":"[9]"},{"why":"Provides the COVID-19 disinformation dataset.","marker":"[14]"},{"why":"Supplies the linguistic and psychological word-count categories used in the climate-change adaptation case study.","marker":"[3]"},{"why":"Gives the emotion-versus-reason interpretation used to explain why Loaded Language and fear appeals are persuasive.","marker":"[8]"}],"fun_headline_variants":["Disinformation tailors persuasion to topic: authority for climate, hypocrisy for war","Domain dictates disinformation's persuasion: repetition, authority, hypocrisy","Study: disinformation's persuasion tactics are domain-specific, not one-size-fits-all","Disinformation picks its persuasion tricks per topic, study finds"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the classifier's labels on the four target datasets are accurate enough that its per-domain differences are real, because the classifier was trained on a different news corpus and is never validated on those datasets.","fun_headline_variants_meta":{"raw":{"variants":["Disinformation tailors persuasion to topic: authority for climate, hypocrisy for war","Domain dictates disinformation's persuasion: repetition, authority, hypocrisy","Study: disinformation's persuasion tactics are domain-specific, not one-size-fits-all","Disinformation picks its persuasion tricks per topic, study finds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.002567,"raw_usage":{"total_tokens":9781,"prompt_tokens":847,"completion_tokens":8934,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":8856}},"tokens_in":463,"tokens_out":8934,"duration_ms":55943,"temperature":1.0,"reasoning_tokens":8856,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:37:12.065188+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually annotate a random sample of sentences from each of the four datasets for the sixteen techniques, compare the classifier's per-domain error rates, and recompute the odds ratios on the human labels; if the domain-specific gaps shrink or vanish, the reported differences are artifacts of uneven misclassification.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Islamic-issues disinformation dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the COVID-19 disinformation dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the linguistic and psychological word-count categories used in the climate-change adaptation case study."}],"review_version":1}