Pith. sign in

REVIEW 3 major objections 6 minor 17 references

Hidden Persuasion: Detecting Manipulative Narratives on Social Media During the 2022 Russian Invasion of Ukraine

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A Gemma 2 classification pipeline with CatBoost post-processing and a dual-head XLM-RoBERTa span detector ranked second and third in the 2025 shared task on manipulation in wartime Ukrainian and Russian Telegram posts.

desk verdict A solid shared-task system description with clear ablations and honest limitations, but the additive-value claim rests on tiny, unvariance-tested F1 deltas. read the letter →

arxiv 2505.24028 v1 pith:CS2WAFFJ submitted 2025-05-29 cs.CL

Kateryna Akhynko , Oleksandr Kosovan , Mykola Trokhymovych This is my paper
classification cs.CL
keywords manipulativenarrativesTelegramUkrainewarmulti-labelclassificationspandetectionLoRAfine-tuningGemma2XLM-RoBERTa
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper reports a competitive system for the 2025 shared task on detecting manipulative narratives in Ukrainian and Russian Telegram posts collected during the 2022 Russian invasion of Ukraine. It claims that a multi-stage pipeline—fine-tuning the Gemma 2 language model with LoRA adapters, stacking a CatBoost classifier on the model's probabilities plus engineered meta-features, and optimizing per-class decision thresholds—placed second in manipulation-technique classification, and that a dual-head XLM-RoBERTa model for joint technique and token classification placed third in span detection. The result matters because Telegram was a primary information source for a large share of Ukrainians during the war, and the paper demonstrates that openly available models fine-tuned with modest compute can flag rhetorical manipulation without relying on fact-checking or user metadata. The paper also documents that performance varies sharply by technique: frequent classes like loaded language are detected reliably, while rare techniques such as straw man and whataboutism remain difficult.

What carries the argument

The load-bearing objects are two trained architectures. For classification, the first stage is a causal-language-model LoRA adapter on the instruction-tuned Gemma 2 2B model, fine-tuned to generate comma-separated lists of manipulation techniques from a prompt that includes technique descriptions and four similarity-selected few-shot examples; the second stage retrains a LoRA adapter in multi-label classification mode; the third stage trains a CatBoost multi-label classifier on the model's probability outputs plus meta-features, namely cosine distances to the centroids of ten K-means clusters of manipulative trigger phrases, frequencies of each technique among the ten nearest training texts and trigger phrases, and surface features such as word count and question marks, followed by per-class thresholds chosen through k-fold cross-validation. For span detection, a dual-head XLM-RoBERTa-Large shares one encoder between a token-level binary classification head and a technique-classification head built from pooled [CLS], mean, and max embeddings, with the auxiliary head's loss down-weighted so that span detection stays primary. The dual-head design is the mechanism by which the paper tests whether joint learning transfers between the two subtasks.

What would settle it

Take the trained pipeline and evaluate it on a time-ordered split, training on earlier Telegram posts and testing on later ones, then compare the resulting macro-F1 against the reported 0.45447 and span F1 against 0.59888; a substantial drop would confirm the chronological-generalization concern and show the system lags behind evolving manipulation tactics.

Watch

Extended reading notes

Core claim

The paper's central claim is that its two-subtask system is among the top performers on the shared task data, reaching a macro-F1 of 0.45447 in technique classification (second place) and a span-level F1 of 0.59888 in span detection (third place). For classification, the authors establish that each stage of their pipeline contributes measurably: the Gemma 2 LoRA fine-tune reaches 0.45007 macro-F1, and the CatBoost post-processing layer with meta-features and per-class thresholds adds about 0.0044. For span detection, they report that the dual-head architecture, which trains a technique-classification head as a weighted auxiliary task alongside the token classifier, improves span F1 from 0.58588 to 0.59888 over the single-head baseline, while noting that the simpler baseline may be more robust for practical use. The authors emphasize that manipulation here is defined by rhetorical and stylistic technique rather than factual falsity, which separates the task from fact-checking.

Load-bearing premise

The system's usefulness in the field rests on the assumption that new, chronologically later Telegram posts will resemble the training posts closely enough for the reported scores to hold, an assumption the paper's own Limitations section flags as untested because the data split ignores post chronology.

Editorial extensions

If this is right

  • The reported gains quantify the value of each design choice: CatBoost post-processing adds roughly 0.0044 macro-F1 (0.45007 to 0.45447), and the dual-head setup adds 0.013 span F1 (0.58588 to 0.59888), so each stage earns its place incrementally.
  • Because the classification pipeline is built on LoRA adapters and an open 2-billion-parameter model, it is reproducible with modest computational resources, and the authors release the code.
  • Per-class threshold optimization prevents under-represented techniques from being overwhelmed by loaded language, which has over ten times the support of the rarest classes.
  • The approach detects manipulation by rhetorical technique, so it complements rather than duplicates fact-checking or knowledge-manipulation detection systems.
  • For span detection, the authors' own comparison suggests that a plain token-classification baseline may be the safer deployment choice, since the dual-head gain is small.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's own limitation note implies that the reported scores likely overstate real-world performance: the train/test split ignores post chronology, so systems tuned to current manipulation patterns may degrade as tactics shift, and re-running the pipeline on a time-ordered split would quantify that gap.
  • The meta-feature recipe, based on cluster distances to trigger phrases and neighbor-label frequencies, is language-agnostic and could transfer to other low-resource manipulation-detection settings beyond Ukrainian and Russian.
  • The auxiliary-head experiment suggests that joint task training helps span detection only marginally; a version that predicts the technique first and conditions span prediction on it, or that trains the auxiliary head at span level rather than post level, is a natural next test.
  • The wide score spread across techniques—0.78 for loaded language against roughly 0.2 to 0.3 for straw man, whataboutism, and bandwagon—implies that deployments focused on influence operations may need dedicated data collection for rare techniques rather than relying on overall macro-F1.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper describes the authors' system for the UNLP 2025 Shared Task on detecting manipulation in social media, covering two subtasks: multi-label classification of manipulation techniques in Ukrainian/Russian Telegram posts and span detection of manipulative text. For classification, they fine-tune Gemma 2 2B IT with two LoRA stages, then combine model probabilities with engineered meta-features in a CatBoost second-level classifier and apply class-wise threshold optimization, reporting second place. For span detection, they use XLM-RoBERTa-Large with a dual-head architecture for joint token-level span classification and technique classification, reporting third place. The paper claims that each enhancement added measurable value, and it closes with a Limitations section acknowledging label imbalance, subjectivity, and the non-chronological train/test split.

Significance. If the reported results are stable, this is a competitive and reasonably practical recipe for multilingual manipulation detection in low-resource settings, and the public code repository and explicit limitation discussion are strengths. The main novel contribution beyond the leaderboard position is the multi-stage pipeline combining generative fine-tuning, meta-features, and threshold optimization, but the current evidence for the additive value of each stage is statistically weak. The paper is therefore of interest to the shared-task and computational-social-science communities, provided the enhancement claims are either substantiated with uncertainty estimates or appropriately softened.

major comments (3)
  1. [Section 6, Tables 2 and 4] The claim that 'each enhancement added measurable value' is not supported by the evidence presented. The classification macro-F1 increases from 0.45007 to 0.45447 (+0.00440) and span F1 from 0.58588 to 0.59888 (+0.01300); these are single point estimates with no standard deviations, confidence intervals, significance tests, or multi-seed runs. Given typical run-to-run variance in fine-tuning, these differences could easily be noise. Please provide repeated runs, bootstrap estimates, or an explicit statement that the improvements are numerical only and may not be statistically significant.
  2. [Sections 4.1 and 4.2] The pipeline is not fully reproducible because key hyperparameters and design choices are missing: LoRA rank/alpha, learning rates, number of epochs, batch size, sequence length, number of folds in the k-fold threshold optimization, and the numerical value of the loss weighting coefficient for the auxiliary classification head. The span-detection baseline is also described only as 'XLM-RoBERTa model configured for token classification' without architecture or training details. These details are necessary for verifying or building on the reported results.
  3. [Sections 5.2 and 6] There is an internal tension between the conclusion that 'each enhancement added measurable value' and the paper's own observation in Section 5.2 that the span-detection gain 'was not large' and that 'for practical applications, a simpler baseline approach may be more robust and justified.' The conclusion should be reconciled with this caveat, or the caveat should be removed from Section 5.2 if the authors maintain that the dual-head gain is meaningful.
minor comments (6)
  1. [References] There are typographical errors in the reference list: 'V orobiov' should be 'Vorobiov' and 'F ourteenth' should be 'Fourteenth'.
  2. [Section 4.1] The phrase 'we used almost the whole training dataset' for the first CLM stage is vague; please specify exactly which portion was held out and how the second-stage training split was constructed.
  3. [Figure 2] The Figure 2 caption says the co-occurrence patterns are shown for training and testing sets together, but the text earlier states that the testing set is split into public and private sets; please clarify whether the figure includes public test labels, private test labels, or both.
  4. [Section 4.2] The sentence 'Consistent with Technique Classification Subtask, we determine optimal prediction thresholds through k-fold cross-validation' is ambiguous for a token-level binary task; specify what threshold is tuned and whether it is applied globally or per token class.
  5. [Section 5.1] The baseline CatBoost model is said to use only meta-features, but the meta-feature set described in Section 4.1 includes cluster distances and neighbor-frequency features computed from training data; please clarify whether the baseline uses all of these meta-features or only simple textual statistics.
  6. [Abstract] The phrase 'trained for multi-target, including token binary classification' is unclear; suggest rewording to 'trained with a multi-task objective combining technique classification and token-level binary span classification'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: this is an empirical shared-task system paper whose claims are evaluated against an external held-out test set, with no definitional or self-citation-based reduction.

full rationale

No circularity found. The paper is an empirical system description for the UNLP 2025 Shared Task, and every headline result (classification macro-F1 0.45447, span F1 0.59888, leaderboard positions) is computed against the external test set defined by the task organizers. The pipeline components—Gemma-2/LoRA two-stage fine-tuning, CatBoost with meta-features, per-class threshold optimization via k-fold cross-validation, and the dual-head XLM-RoBERTa architecture—are standard supervised-learning procedures. Parameters, thresholds, and post-processing are fit only on the training split, and the final predictions are evaluated on held-out test data; there is no quantity that is defined in terms of the metric it is said to predict. The comparison between the baseline and the proposed systems (Tables 2 and 4) is an empirical ablation, not a derivation. The small F1 differences are a legitimate statistical-robustness concern regarding the claim that 'each enhancement added measurable value,' but that concern is about variance and significance testing, not about circularity: the deltas are not forced by construction. The paper's self-citations (Trokhymovych and Saez-Trumper 2021; Trokhymovych et al. 2024, 2025) appear only in related-work context and are not load-bearing for the system's performance or for any uniqueness claim. The Limitations paragraph explicitly acknowledges that the non-chronological train/test split may overestimate real-world performance; that is an external-validity caveat rather than a circular step. Overall, the derivation chain is self-contained and externally validated, so the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The system's performance depends on a set of hyperparameters and assumptions. The paper provides no formal derivation, so the ledger lists the main tuned hyperparameters and the domain assumptions the evaluation relies on. No invented entities are introduced.

free parameters (5)
  • Number of clusters K in K-means = 10
    Set equal to the number of manipulation techniques (Section 4.1); no data-driven justification given.
  • Number of few-shot examples per prompt = 4
    Two selected by text similarity and two by trigger-phrase similarity (Section 4.1); no ablation or explanation of this count.
  • Class-wise prediction thresholds = Not reported
    Optimized via k-fold cross-validation with median selection (Section 4.1); these thresholds directly affect the final macro-F1.
  • LoRA hyperparameters = Not reported
    Rank, alpha, learning rate, and number of epochs for both Gemma fine-tuning stages are not stated, hindering reproduction.
  • Loss weighting coefficient for auxiliary classification head = Not reported
    The paper mentions a 'reduced weighting coefficient' for the classification head loss (Section 4.2) but never gives its value.
assumptions (3)
  • domain assumption The shared task annotations (labels and spans) are treated as ground truth.
    The evaluation depends on these annotations being correct; the paper itself notes in Limitations that the manipulation signal is subjective and context-dependent.
  • domain assumption The train/test split is representative of a real deployment scenario.
    The paper's Limitations section admits the split ignores chronological order, so the models may not generalize to new manipulation patterns.
  • domain assumption Pretrained multilingual models transfer to informal Ukrainian/Russian Telegram text.
    The approach relies on Gemma 2 and XLM-RoBERTa, which have limited support for these languages and informal text, as the Limitations section acknowledges.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hidden Persuasion: Detecting Manipulative Narratives on Social Media During the 2022 Russian Invasion of Ukraine." pith.science (2026). https://pith.science/paper/CS2WAFFJ

@misc{pith2026250524028,
  author       = {Pith},
  title        = {Pith review of: Hidden Persuasion: Detecting Manipulative Narratives on Social Media During the 2022 Russian Invasion of Ukraine},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CS2WAFFJ}},
  note         = {Machine review of arXiv:2505.24028}
}
read the original abstract

This paper presents one of the top-performing solutions to the UNLP 2025 Shared Task on Detecting Manipulation in Social Media. The task focuses on detecting and classifying rhetorical and stylistic manipulation techniques used to influence Ukrainian Telegram users. For the classification subtask, we fine-tuned the Gemma 2 language model with LoRA adapters and applied a second-level classifier leveraging meta-features and threshold optimization. For span detection, we employed an XLM-RoBERTa model trained for multi-target, including token binary classification. Our approach achieved 2nd place in classification and 3rd place in span detection.

Figures

Figures reproduced from arXiv: 2505.24028 by the authors.

Figure 1
Figure 1. Sketch of manipulation techniques classifica [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Co-occurrence of manipulation techniques in [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Pipeline of technique classification solution [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Pipeline of span identification solution [PITH_FULL_IMAGE:figures/full_fig_p004_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

17 extracted references · 6 canonical work pages

  1. [1]

    Ola Altiti, Malak Abdullah, and Rasha Obiedat. 2020. https://doi.org/10.18653/v1/2020.semeval-1.229 JUST at S em E val-2020 task 11: Detecting propaganda techniques using BERT pre-trained model . In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1749--1755, Barcelona (online). International Committee for Computational Linguistics

    work page doi:10.18653/v1/2020.semeval-1.229 2020

  2. [2]

    Parishad BehnamGhader, Vaibhav Adlakha, Marius Mosbach, Dzmitry Bahdanau, Nicolas Chapados, and Siva Reddy. 2024. https://arxiv.org/abs/2404.05961 Llm2vec: Large language models are secretly powerful text encoders . Preprint, arXiv:2404.05961

    arXiv 2024

  3. [3]

    Giovanni Da San Martino, Alberto Barr \'o n-Cede \ n o, Henning Wachsmuth, Rostislav Petrov, and Preslav Nakov. 2020. https://doi.org/10.18653/v1/2020.semeval-1.186 S em E val-2020 task 11: Detection of propaganda techniques in news articles . In Proceedings of the Fourteenth Workshop on Semantic Evaluation, pages 1377--1414, Barcelona (online). Internati...

    doi:10.18653/v1/2020.semeval-1.186 2020

  4. [4]

    Artur Kiulian, Anton Polishko, Mykola Khandoga, Oryna Chubych, Jack Connor, Raghav Ravishankar, and Adarsh Shirawalmath. 2024. https://arxiv.org/abs/2404.09138 From bytes to borsch: Fine-tuning gemma and mistral for the ukrainian language representation . Preprint, arXiv:2404.09138

    arXiv 2024

  5. [5]

    Rahul Mehta and Vasudeva Varma. 2023. https://doi.org/10.18653/v1/2023.semeval-1.62 LLM - RM at S em E val-2023 task 2: Multilingual complex NER using XLM - R o BERT a . In Proceedings of the 17th International Workshop on Semantic Evaluation (SemEval-2023), pages 453--456, Toronto, Canada. Association for Computational Linguistics

    work page doi:10.18653/v1/2023.semeval-1.62 2023

  6. [6]

    Nils Reimers and Iryna Gurevych. 2019. https://arxiv.org/abs/1908.10084 Sentence-bert: Sentence embeddings using siamese bert-networks . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics

    arXiv 2019

  7. [7]

    Jacob, Sakshi Kalra, and Yashvardhan Sharma

    Siva Sai, Alfred W. Jacob, Sakshi Kalra, and Yashvardhan Sharma. 2021. Stacked embeddings and multiple fine-tuned xlm-roberta models for enhanced hostility identification. In Combating Online Hostile Posts in Regional Languages during Emergency Situation, pages 224--235, Cham. Springer International Publishing

    work page 2021

  8. [8]

    Veronika Solopova, Christoph Benzm \"u ller, and Tim Landgraf. 2023. https://doi.org/10.18653/v1/2023.unlp-1.5 The evolution of pro-kremlin propaganda from a machine learning and linguistics perspective . In Proceedings of the Second Ukrainian Natural Language Processing Workshop (UNLP), pages 40--48, Dubrovnik, Croatia. Association for Computational Linguistics

    doi:10.18653/v1/2023.unlp-1.5 2023

Show all 17 references
  1. [9]

    Kilian Sprenkamp, Daniel Gordon Jones, and Liudmila Zavolokina. 2023. https://arxiv.org/abs/2310.06422 Large language models for propaganda detection . Preprint, arXiv:2310.06422

    2023 arXiv

  2. [10]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024 arXiv

  3. [11]

    Mykola Trokhymovych, Oleksandr Kosovan, Nathan Forrester, Pablo Aragón, Diego Saez-Trumper, and Ricardo Baeza-Yates. 2025. https://arxiv.org/abs/2504.10663 Characterizing knowledge manipulation in a russian wikipedia fork . Preprint, arXiv:2504.10663

    2025 arXiv

  4. [12]

    Mykola Trokhymovych and Diego Saez-Trumper. 2021. https://doi.org/10.1145/3459637.3481961 Wikicheck: An end-to-end open source automatic fact-checking api based on wikipedia . In Proceedings of the 30th ACM International Conference on Information and Knowledge Management, CIKM...

    2021

  5. [13]

    Mykola Trokhymovych, Indira Sen, and Martin Gerlach. 2024. https://doi.org/10.18653/v1/2024.acl-long.342 An open multilingual system for scoring readability of W ikipedia . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Lo...

    2024 doi

  6. [14]

    Mykyta Vorobiov. 2024. https://cepa.org/article/has-ukraine-become-too-dependent-on-telegram/ Has ukraine become too dependent on telegram? Accessed: 12 April 2025

    2024

  7. [15]

    Xin Zhang, Yanzhao Zhang, Dingkun Long, Wen Xie, Ziqi Dai, Jialong Tang, Huan Lin, Baosong Yang, Pengjun Xie, Fei Huang, et al. 2024. mgte: Generalized long-context text representation and reranking models for multilingual text retrieval. In Proceedings of the 2024 Conference ...

    2024

  8. [16]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  9. [17]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.