REVIEW 5 major objections 5 minor 3 references
MultiProSE: A Multi-label Arabic Dataset for Propaganda, Sentiment, and Emotion Detection
T0 review · 5 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read MultiProSE is the first Arabic dataset to label the same 8,000 news articles for propaganda, sentiment, and emotion, and the paper reports baseline results for all three tasks.
desk verdict A genuine Arabic propaganda-sentiment-emotion annotation effort, but the guidelines impose the sentiment-emotion correlation it claims to discover, and the placeholder release makes it unverifiable. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is MultiProSE itself, an 8,000-text corpus in which every Modern Standard Arabic news paragraph carries three labels: propaganda (true/false, inherited from ArPro), sentiment (positive/negative/neutral), and emotion (happiness/sadness/anger/fear/none). The mechanism that makes it a usable benchmark is the annotation protocol: three paid native-speaker annotators with doctoral training, a gold-data quality-control phase with a 70% trust threshold, a qualifying exam, majority voting, and a consolidation phase that can add up to six annotators. The guidelines encode a mapping from sentiment to emotion, pairing positive with happiness and negative with anger, sadness, or fear, so the protocol both produces the labels and shapes how the dimensions relate to each other.
What would settle it
Inspect the released sentiment-by-emotion contingency table: if almost every positive text is labeled happiness and almost every negative text is labeled anger, sadness, or fear, the two dimensions are not independent and the reported propaganda-emotion links largely reflect the annotation rule. A direct check is to re-annotate a random sample with fresh annotators who receive no sentiment-emotion mapping and compare the resulting agreement and correlations.
Extended reading notes
Core claim
The paper's central claim is that a manually annotated Arabic news corpus can carry propaganda, sentiment, and emotion labels on every text and still be annotated reliably. MultiProSE extends ArPro's 8,000 paragraphs by adding sentiment and emotion labels under a schema adapted from a six-basic-emotions model, dropping disgust and adding a 'none' category. The authors report averaged pairwise kappa values of 0.7074-0.8128 and multi-annotator kappa values of 0.7093-0.7650, which they read as substantial agreement. Baseline results show GPT-4o-mini reaching Micro-F1 scores of 0.842 for sentiment and 0.750 for emotion, while AraBERT and GPT-4o-mini both reach 0.769 for propaganda. The authors also report that propaganda is more frequent in negative and positive sentiment texts than in neutral ones, with anger and happiness the most common emotions among propagandistic paragraphs.
Load-bearing premise
The load-bearing premise is that sentiment and emotion can be annotated as independent dimensions even though the guidelines tell annotators to pair positive sentiment with happiness and negative sentiment with anger, sadness, or fear; if annotators follow that rule, the dataset's sentiment-emotion correlations are built in rather than discovered.
Editorial extensions
If this is right
- MultiProSE provides a public Arabic benchmark with fixed train/test splits for propaganda detection, sentiment analysis, and emotion recognition, allowing direct comparison of future models.
- The baselines set reference numbers: GPT-4o-mini reaches Micro-F1 0.842 for sentiment and 0.750 for emotion, and AraBERT ties GPT-4o-mini at 0.769 for propaganda.
- The corpus enables analysis of how opinion dimensions co-occur, with the reported data showing propaganda concentrated in negative and positive sentiment and most often paired with anger or happiness.
- Because the dataset, guidelines, and code are released, researchers can extend the work to span-level annotation or Arabic sentiment and emotion lexicons, as the paper suggests for future work.
Reading between the lines
- Editorial extension: the paper does not test its motivating claim that sentiment and emotion features improve propaganda detection; the natural next experiment is to feed sentiment and emotion predictions into the propaganda classifier and measure the gain.
- Editorial extension: since the propaganda labels come from ArPro, MultiProSE's novelty is the added sentiment and emotion layers, and the 'largest propaganda dataset' claim should be read in that light rather than as a new collection of propaganda annotations.
- Editorial extension: the sentiment-emotion mapping written into the guidelines means the interaction distributions in Figure 3 should be re-examined with fresh annotators who are not given that mapping, to see whether the correlations persist.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces MultiProSE, an Arabic corpus of 8,000 news paragraphs that extends the existing ArPro propaganda dataset with manually annotated sentiment and emotion labels. It describes the annotation protocol, a gold-data quality-control mechanism, inter-annotator agreement results, label distributions, and baseline experiments with AraBERT, XLM-RoBERTa, and GPT-4o-mini. The stated contributions are a new Arabic benchmark for propaganda detection, sentiment analysis, and emotion recognition, together with an analysis of how these opinion dimensions interact in news text.
Significance. If the dataset is actually released and the annotation caveats are resolved, MultiProSE would be a useful Arabic multi-task benchmark. The design includes three paid native-speaker annotators with doctoral degrees, a pre-exam and gold-data phase, and reported inter-annotator agreement in the substantial range (Light's kappa 0.7074-0.8128; Fleiss' kappa 0.7093-0.7650), which are genuine strengths. The baseline results from multiple-seed runs also provide useful reference points. However, the manuscript currently overstates the dataset size, presents inconsistent train/test splits, uses a placeholder repository link, and, most importantly, describes an annotation guideline that deterministically couples sentiment and emotion labels while claiming these dimensions are annotated independently. These issues must be corrected before the benchmark and interaction-analysis claims can be accepted.
major comments (5)
- [3.3 / Figure 3] Section 3.3 states that a positive text will have happiness as its emotion label, that a negative text will be labeled anger, sadness, or fear, and that these details were 'incorporated' into the guidelines; the same section claims this 'ensure[s] that sentiment is annotated independently' of emotion. These statements are contradictory. If annotators followed the guidelines, sentiment and emotion are coupled by the annotation schema, so the co-occurrence patterns in Figure 3 and any sentiment-emotion interaction analysis are partly products of the design rather than discoveries about the text. Please re-annotate with independent dimensions, or substantially re-frame the claims and report the analyses conditional on the enforced mapping.
- [Abstract / Table 1] The abstract's statement that 8,000 articles make MultiProSE 'the largest propaganda dataset to date' is contradicted by Table 1, which lists TSHP-17 with 22,580 articles and QProp with 51,294 articles. If the intended claim is 'largest Arabic manually annotated propaganda dataset,' it should be stated precisely and supported; otherwise the global claim should be removed.
- [3.1 / 3.7 / 4.1] The dataset split is reported inconsistently: Section 3.1 gives 6,002/672/1,326 for train/validation/test; Section 3.7 gives 6,680/1,320 train/test with no validation set; Section 4.1 says 75%/8.5%/16.5%. These numbers cannot all describe the released data, and the discrepancy directly affects the comparability of the baseline results. Specify the exact released split and use it consistently throughout.
- [Footnote 1] The availability statement in the Abstract is not currently verifiable: footnote 1 reads 'https://github.com/xxx/xxx', which is a placeholder. For a dataset paper the repository URL, the dataset, and the annotation guidelines must be accessible; please provide the actual link and confirm the license.
- [Title / 3.7] The term 'multi-label' is used for the dataset, but each text receives a single propaganda label, a single sentiment label, and a single emotion label. This is multi-task, not multi-label in the standard sense. If the intended meaning is that each text carries several label dimensions, please state this explicitly; if some tasks are multi-label (e.g., multiple propaganda techniques per text), then the annotation and evaluation sections need to reflect that.
minor comments (5)
- [3.2] The majority-voting description is unclear: if three annotators disagree, the text says a sixth annotator may be added, but with three annotators majority voting already produces a decision; please specify the conflict-resolution rule and when and how additional annotators are consulted.
- [3.6] The text refers to both 'Light's Kappa' and 'Lights' index'; please standardize the name of the measure to Light's kappa throughout.
- [4.2] The model ArabicBERT is mentioned in the results discussion but is not introduced in Section 4.1, which describes only AraBERT and XLM-RoBERTa; please define it or remove the reference.
- [3.3] The citation to MELD [38] is misleading here because MELD does not enforce a deterministic sentiment-emotion mapping; please make the relationship precise or use a different justification.
- [3.5] The sentence about 12 batches and 24,000 collected annotations follows from 8,000 texts times three annotations, but the relation to the reported rounds and batches should be stated explicitly to avoid confusion.
Circularity Check
Annotation guidelines encode a fixed sentiment-to-emotion mapping, making the reported sentiment-emotion correlations partly definitional; the propaganda dimension remains independent.
-
self definitional
[Section 3.3 (Annotation Guidelines), interpreted in Section 3.7 and Figure 3.]
"As mentioned by the authors [38], the labeling of sentiment is correlated with emotion labels. This means that a text annotated as positive will have happiness as the emotion label, while a text with negative sentiment will be annotated with anger, sadness, or fear as emotion labels. Therefore, in the annotation guidelines, these details have been considered and incorporated to ensure that sentiment is annotated independently of the emotions in the text and vice versa."
The guidelines instruct annotators to assign happiness to positive texts and anger, sadness, or fear to negative texts. This imposes a sentiment-emotion mapping by construction rather than letting the two dimensions vary freely. The paper then presents the resulting label distribution in Figure 3 and claims that the dataset enables studying how opinion dimensions interact. The sentiment-emotion margin is therefore not an empirical discovery of the corpus but a direct consequence of the annotation schema. The sentence claiming this ensures sentiment is 'annotated independently' is internally contradictory: a fixed mapping cannot produce independence.
full rationale
This is a dataset-release paper, not a derivation paper: there is no fitted parameter being renamed as a prediction and no self-citation chain that forces the central result. The ArPro propaganda labels are taken from independent prior work, and the LLM/BERT baselines are evaluated on an external test split, so the benchmark itself is self-contained. The one substantive circularity is in the annotation design. Section 3.3 explicitly incorporates a correlation between sentiment and emotion into the guidelines: positive maps to happiness, negative maps to anger, sadness, or fear. Because annotators were told to apply this mapping, the sentiment-emotion correlations visible in the dataset statistics are guaranteed by the schema, not discovered from the text. The paper's contribution statement and conclusion emphasize analyzing how various opinion dimensions interact, which makes this schema-imposed coupling load-bearing for part of the claimed scientific value. The propaganda-sentiment and propaganda-emotion distributions, by contrast, combine a pre-existing propaganda label with new sentiment and emotion labels and are not circular. Overall, the central dataset resource remains independently valuable, but the specific claim that sentiment and emotion are annotated independently and that their interaction is an empirical finding is compromised by construction.
Assumptions & free parameters
assumptions (5)
- domain assumption Ekman's basic emotion model is valid for Arabic news, and disgust can be omitted by merging it with anger.
- domain assumption Sentiment and emotion labels are correlated, and this correlation can be incorporated into guidelines while still claiming independent annotation.
- domain assumption The ArPro propaganda labels are correct and can be reused as ground truth.
- domain assumption Manual annotation by three PhD-level native Arabic speakers with majority voting yields reliable ground truth.
- domain assumption Modern Standard Arabic news paragraphs, with a small amount of dialectal content, are representative of Arabic propaganda.
Cite this review
Pith. "Pith review of MultiProSE: A Multi-label Arabic Dataset for Propaganda, Sentiment, and Emotion Detection." pith.science (2026). https://pith.science/paper/XM227B42
@misc{pith2026250208319,
author = {Pith},
title = {Pith review of: MultiProSE: A Multi-label Arabic Dataset for Propaganda, Sentiment, and Emotion Detection},
year = {2026},
howpublished = {\url{https://pith.science/paper/XM227B42}},
note = {Machine review of arXiv:2502.08319}
}
read the original abstract
Propaganda is a form of persuasion that has been used throughout history with the intention goal of influencing people's opinions through rhetorical and psychological persuasion techniques for determined ends. Although Arabic ranked as the fourth most-used language on the internet, resources for propaganda detection in languages other than English, especially Arabic, remain extremely limited. To address this gap, the first Arabic dataset for Multi-label Propaganda, Sentiment, and Emotion (MultiProSE) has been introduced. MultiProSE is an open-source extension of the existing Arabic propaganda dataset, ArPro, with the addition of sentiment and emotion annotations for each text. This dataset comprises 8,000 annotated news articles, which is the largest propaganda dataset to date. For each task, several baselines have been developed using large language models (LLMs), such as GPT-4o-mini, and pre-trained language models (PLMs), including three BERT-based models. The dataset, annotation guidelines, and source code are all publicly released to facilitate future research and development in Arabic language models and contribute to a deeper understanding of how various opinion dimensions interact in news media1.
Reference graph
Works this paper leans on
-
[10]
COVID-19 in Bulgarian Social Media: Factuality, Harmfulness, Propaganda, and Framing,
Qatar Computing Research Institute, HBKU, Qatar et al., “COVID-19 in Bulgarian Social Media: Factuality, Harmfulness, Propaganda, and Framing,” in Proceedings of the Conference Recent Advances in Natural Language Processing - Deep Learning for Natural Language Processing Methods and Applications, INCOMA Ltd. Shoumen, BULGARIA, 2021, pp. 997–1009. doi: 10....
-
[31]
Arabic language sentiment analysis on health services,
A. M. Alayba, V. Palade, M. England, and R. Iqbal, “Arabic language sentiment analysis on health services,” in 2017 1st International Workshop on Arabic Script Analysis and Recognition (ASAR), Nancy, France: IEEE, Apr. 2017, pp. 114–118. doi: 10.1109/ASAR.2017.8067771. [32] N. Al-Twairesh, “Sentiment Analysis of Twitter: A Study on the Saudi Community,” P...
-
[46]
Comparing BERT against traditional machine learning text classification,
S. González-Carvajal and E. C. Garrido-Merchán, “Comparing BERT against traditional machine learning text classification,” JCCE, vol. 2, no. 4, pp. 352–356, Apr. 2023, doi: 10.47852/bonviewJCCE3202838. [47] W. Antoun, F. Baly, and H. Hajj, “AraBERT: Transformer-based Model for Arabic Language Understanding,” arXiv:2003.00104 [cs], Mar. 2021, Accessed: May...
arXiv 2023
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.