Pith. sign in

REVIEW 4 major objections 6 minor 23 references

EmoHopeSpeech: An Annotated Dataset of Emotions and Hope Speech in English and Arabic

T0 review · 4 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A bilingual Arabic-English dataset links emotion labels to hope speech.

desk verdict Potentially useful bilingual dataset, but the submitted abstract's reliability numbers contradict the body's own tables, and the paper cannot currently support its central claim. read the letter →

arxiv 2505.11959 v2 pith:U23S4IS5 submitted 2025-05-17 cs.CL

Wajdi Zaghouani , Md. Rafiul Biswas This is my paper · ORCID
classification cs.CL
keywords emotionrecognitionhopespeechdetectionArabicNLPbilingualdatasetannotationreliabilityFleisskappacross-linguisticanalysissocialmediatext
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

EmoHopeSpeech is a new resource of 27,456 Arabic and 10,036 English social-media posts, each carrying a basic emotion label plus deeper annotations: emotion intensity, complexity, cause, and hope-speech category with subcategories. The authors aim to provide the first large English-Arabic dataset that supports joint analysis of emotion and hope speech, addressing what they see as a gap for Arabic in particular. If the resource works as claimed, it would let researchers compare how positivity and emotion are expressed across languages, and give model builders a training ground for detecting uplifting rather than only toxic content. The paper validates the resource through annotator agreement and baseline classifiers, and its abstract reports high validation scores.

What carries the argument

The load-bearing instrument is the multi-axis annotation scheme, which records emotion and hope speech as layered structures rather than a single sentiment score. It is what lets the paper compute relationships between emotion characteristics and hope categories, and it defines the gold standard on which the baseline models are trained. The evaluation machinery pairs that scheme with Fleiss' kappa computed on a shared sample of 200 instances, plus logistic regression, multinomial naive Bayes, and fine-tuned AraBERT and BERT classifiers for hope-speech prediction.

What would settle it

Recompute Fleiss' kappa on the full released Arabic and English files using the paper's own category definitions. If the full-corpus values match the body's 0.31-0.56 range rather than the abstract's 0.75-0.85, the reliability claim is contradicted; likewise, re-running the best classifier on an independently re-annotated sample should come near the abstract's F1 of 0.69 if that validation claim is correct.

Watch

Extended reading notes

Core claim

The central claim is that the paper has built the first large-scale bilingual dataset combining emotion and hope speech annotations in English and Arabic. The contribution is the annotation scheme itself: each text receives an emotion label, an intensity level, a complexity level, a cause category, a hope-speech category (hope, not hope, counter, neutral, hate/negativity), and a hope subcategory (inspirational, solidarity, resilience, spiritual). The paper argues that the annotations are reliable enough to support machine-learning detection, and that statistically significant chi-square associations between emotion features and hope categories--for example emotion cause versus hope category ($\chi^2=810.3$ for Arabic and $245.6$ for English)--show the two phenomena are genuinely coupled in the data.

Load-bearing premise

The load-bearing premise is that the validation figures in the abstract--Fleiss kappa 0.75-0.85 and F1 0.69--are the correct ones, while the body reports kappa 0.31-0.56 and F1 no higher than 0.47; if the abstract's numbers are wrong, the claim that the annotations are trustworthy collapses.

Editorial extensions

If this is right

  • Cross-linguistic comparisons become concrete: the paper reports that Arabic skews toward medium-complexity and internal-reflection emotions, while English skews toward simple complexity and low intensity.
  • Joint emotion-hope modeling becomes feasible because emotion cause is strongly associated with hope category in both languages, giving models a predictive signal beyond the hope label itself.
  • The four hope subcategories turn hope detection from a binary task into a finer-grained classification, which is useful for applications that want to distinguish encouragement, solidarity, resilience, and spiritual support.
  • The body's baseline results (best F1 of 0.47 for Arabic and 0.45 for English) give later work concrete numbers to beat rather than a claim that the task is solved.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the body's agreement values (Fleiss kappa 0.31-0.56) are below the abstract's (0.75-0.85), so a cautious user should treat the main hope-speech category as the most dependable label and the emotion-cause and subcategory labels as exploratory.
  • Because emotion cause is the strongest correlate of hope category, a natural follow-up is a cause-aware model that predicts hope speech using emotion-cause labels, testing whether it beats the paper's plain baselines.
  • The large distributional differences between the two languages (e.g., 80% neutral labels in English versus 13% in Arabic) may reflect the source corpora as much as cultural expression; a rigorous cross-cultural comparison would require matched sampling across genres.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents EmoHopeSpeech, a bilingual (Arabic/English) corpus annotated for emotion intensity, emotion complexity, emotion cause, and hope speech categories with subcategories. The authors describe data collection from existing emotion datasets, filtering, annotation by native speakers, inter-annotator agreement analysis, chi-square association tests between emotion features and hope speech, and baseline classification experiments with logistic regression, naive Bayes, and transformer models (AraBERT/BERT). The central claims are that the dataset is large and that its annotations are reliable, supported by the abstract's reported Fleiss' Kappa of 0.75–0.85 and F1-score of 0.69.

Significance. If the reported reliability and size figures were accurate, EmoHopeSpeech would be a valuable bilingual resource for affective computing, particularly for Arabic, and the public release via Zenodo and GitHub is a positive feature. The inclusion of multiple annotation layers (intensity, complexity, cause, hope subcategories) addresses a genuine gap. However, as submitted, the manuscript contains direct contradictions between the abstract and the body's evaluation tables, internal inconsistencies in the dataset statistics, and a circular validation argument. These issues affect the central claim of a validated large-scale resource, so the paper's contribution as currently stated is not established. The dataset may still be useful to the community, but the claims need to be substantially corrected and the validation strengthened.

major comments (4)
  1. [Abstract; §6, Table 12; §5.2, Table 11] The abstract states that Fleiss' Kappa revealed 0.75–0.85 agreement and that baseline F1-score 0.69 validates the annotations, but Table 12 reports kappas of 0.31–0.56 and Table 11 reports F1 scores no higher than 0.47. The paper's own conclusion (Section 7) characterizes the kappas as 'moderate.' Since annotation reliability is the central claim of a dataset paper, this direct contradiction must be resolved: either the body's tables are wrong or the abstract's figures are unsupported. The manuscript cannot be accepted with both sets of numbers standing.
  2. [§3.1; Abstract] Section 3.1 states that of 10,036 English rows remaining after filtering, only 4,036 were annotated and included in the study, yet the abstract reports '10,036 entries for English' as part of the dataset. Table 9's English counts sum to 4,036, confirming that the released/analyzed English resource is 4,036 rows, not 10,036. The abstract overstates the dataset size by a factor of 2.5, which is a load-bearing misrepresentation of the resource.
  3. [§5.2] The sentence 'This evaluation not only validated the robustness of the annotation process' uses model performance on the same annotated pool as evidence of label quality. Because the models are trained and evaluated on annotations produced by the same process, high or low F1 does not independently validate the annotations; it only measures learnability under the annotators' labels. An external gold subset, a comparison with existing benchmark labels, or at minimum a clear train/test split and discussion of the ceiling imposed by the measured kappa is needed to support the reliability claim.
  4. [§4, Tables 7–9] The dataset statistics are internally inconsistent: Table 7 English emotion counts sum to 4,031 while Table 9 English hope-speech category counts sum to 4,036; Table 8's English complexity, intensity, and cause counts sum to 3,946, 3,980, and 3,987 respectively. These discrepancies are unexplained and undermine the statistical analyses in Section 5.1, which rely on the same counts. The authors should recompute and reconcile all descriptive statistics against the released data.
minor comments (6)
  1. [Abstract (first page)] The abstract printed at the top of the paper reports '23,456 entries for Arabic,' while the full-text abstract and Section 3.1 report 27,456; this should be corrected to a single number.
  2. [Table 11] In the English hope-speech prediction block, the model is labeled 'AraBERT,' but the text says BERT-based-uncased was used for English; this appears to be a copy-paste error.
  3. [§6] The Fleiss' Kappa calculation is described only as being on 'a sample of 200 instances'; the selection procedure for this sample and whether it is representative of the full corpus should be stated.
  4. [§5.1] The heading 'Corelation' is misspelled; it should be 'Correlation.'
  5. [§2] The in-text citation 'Divakaran, Girish, and Shashirekha' is missing a year; the reference list gives 2024, so the citation should include it.
  6. [§9] The dataset release paragraph is grammatically incomplete ('We have made dataset publicly available to the 10.5281/zenodo.14669301'); it should provide the full URL and say 'available at' instead of 'to the.'

Circularity Check

1 steps flagged · score 4.0 of 10

Dataset-quality claim rests on self-referential model evaluation; abstract's reliability figures contradict body's tables.

  1. fitted input called prediction [Abstract and Section 5.2 (Dataset Evaluation), Tables 11 and 12]
    "The evaluation metrics (F1-score: 0.69) obtained from the baseline model (i.e., machine learning model) validate that the data annotations are worthy... This evaluation not only validated the robustness of the annotation process but also underscored the role of advanced models in enhancing predictive accuracy for hope speech detection tasks."

    The baseline model is trained on the same annotator-produced labels that it is then used to 'validate.' The F1 score measures how well the model can recover the annotations it was trained to predict, not whether those annotations correspond to any independent ground truth. Calling this performance a validation of annotation robustness closes the loop: the annotations are both the supervision signal and the evaluation reference. No external benchmark or human-verified gold subset is introduced. The only independent evidence of annotation quality would be the annotator agreement scores, but Table 12 reports Fleiss' Kappa of 0.31-0.56, contradicting the abstract's 0.75-0.85.

full rationale

The paper is primarily a dataset contribution, and its new annotations are not derived from a mathematical model, so most of the construction pipeline is not circular. The clearest circular step is the validation logic: a model is fitted to the annotated dataset and its agreement with those same annotations is presented as evidence that the annotations are reliable. That is statistically self-referential and does not validate label quality beyond the annotator-agreement scores, which are themselves modest (0.31-0.56 in Table 12) and inconsistent with the abstract's claimed 0.75-0.85. The dataset-size claim also conflicts with Section 3.1, which says only 4,036 of the 10,036 English rows were annotated. These are correctness and consistency problems rather than derivation-circularity, but they compound the weakness of the validation argument. The self-citation of the authors' earlier Arabic hate-speech dataset is only one of three data sources and is not load-bearing, so it does not raise the score. Overall, the central reliability claim is partially circular and additionally undermined by internal numerical contradictions, warranting a moderate score of 4.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No mathematical derivation is present; the paper's contributions are data and annotations. The load-bearing inputs are the source datasets, the annotation protocol, and the assumption that model F1 on the same annotation pool indicates annotation quality. The abstract's reliability numbers are contradicted by the body.

assumptions (3)
  • domain assumption Fleiss' Kappa is the correct reliability measure for the multi-annotator categorical setup.
    Section 6 uses Fleiss' Kappa without discussing alternative agreement measures or whether the categories are nominal and exclusive. The low values for subjective labels make this choice consequential.
  • ad hoc to paper A model trained and evaluated on the same annotation pool can validate annotation quality.
    Section 5.2 states the evaluation 'validated the robustness of the annotation process'. This premise is assumed, not demonstrated, and is undermined by the weak F1 scores in Table 11.
  • domain assumption Pre-existing emotion labels from the three Arabic source datasets are trustworthy enough to serve as gold labels.
    Section 3.1 merges 49,517 rows from three prior datasets with no reported audit of label quality or compatibility of label schemas.

how reviews work

0 comments
Cite this review

Pith. "Pith review of EmoHopeSpeech: An Annotated Dataset of Emotions and Hope Speech in English and Arabic." pith.science (2026). https://pith.science/paper/U23S4IS5

@misc{pith2026250511959,
  author       = {Pith},
  title        = {Pith review of: EmoHopeSpeech: An Annotated Dataset of Emotions and Hope Speech in English and Arabic},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/U23S4IS5}},
  note         = {Machine review of arXiv:2505.11959}
}
read the original abstract

This research introduces a bilingual dataset comprising 23,456 entries for Arabic and 10,036 entries for English, annotated for emotions and hope speech, addressing the scarcity of multi-emotion (Emotion and hope) datasets. The dataset provides comprehensive annotations capturing emotion intensity, complexity, and causes, alongside detailed classifications and subcategories for hope speech. To ensure annotation reliability, Fleiss' Kappa was employed, revealing 0.75-0.85 agreement among annotators both for Arabic and English language. The evaluation metrics (micro-F1-Score=0.67) obtained from the baseline model (i.e., using a machine learning model) validate that the data annotations are worthy. This dataset offers a valuable resource for advancing natural language processing in underrepresented languages, fostering better cross-linguistic analysis of emotions and hope speech.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

23 extracted references · 16 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Al-Khatib, A.; and El-Beltagy, S. R. 2018. Emotional tone detection in arabic tweets. In Computational Linguistics and Intelligent Text Processing: 18th International Conference, CICLing 2017, Budapest, Hungary, April 17--23, 2017, Revised Selected Papers, Part II 18, 105--114. Springer

    work page 2018

  4. [4]

    Alrefaie, M. T. 2019. Arabic Stop Words

    work page 2019

  5. [5]

    Anjali, S. 2024. Emotion analysis based on text

    work page 2024

  6. [6]

    Antoun, W.; Baly, F.; and Hajj, H. 2020. Arabert: Transformer-based model for arabic language understanding. arXiv preprint arXiv:2003.00104

    arXiv 2020

  7. [7]

    Babanne, V.; Borgaonkar, M.; Katta, M.; Kudale, P.; and Deshpande, V. 2020. Emotion based personalized recommendation system. Int. Res. J. Eng. Technol.(IRJET), 7: 701--705

    work page 2020

  8. [8]

    Balouchzahi, F.; Sidorov, G.; and Gelbukh, A. 2023. Polyhope: Two-level hope speech detection from tweets. Expert Systems with Applications, 225: 120078

    work page 2023

Show all 23 references
  1. [9]

    Chakravarthi, B. R. 2020. HopeEDI: A multilingual hope speech detection dataset for equality, diversity, and inclusion. In Proceedings of the Third Workshop on Computational Modeling of People's Opinions, Personality, and Emotion's in Social Media, 41--53

    2020

  2. [10]

    N.; Joshi, M.; and Agrawal, P

    Chatterjee, A.; Narahari, K. N.; Joshi, M.; and Agrawal, P. 2019. S em E val-2019 Task 3: E mo C ontext Contextual Emotion Detection in Text. In Proceedings of the 13th International Workshop on Semantic Evaluation, 39--48. Minneapolis, Minnesota, USA: Association for Computat...

    2019

  3. [11]

    Chintala, S. 2024. Emotion AI in Business Intelligence: Under-standing Customer Sentiments and Behaviors. Central Asian Journal of Mathematical Theory and Com-puter Sciences, 5(3): 205--212

    2024

  4. [12]

    Devlin, J.; Chang, M.; Lee, K.; and Toutanova, K. 2018. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. CoRR, abs/1810.04805

    2018 arXiv

  5. [13]

    Dheeraj, K.; and Ramakrishnudu, T. 2021. Negative emotions detection on online mental-health related patients texts using the deep learning with MHA-BCNN model. Expert Systems with Applications, 182: 115265

    2021

  6. [14]

    Divakaran, S.; Girish, K.; and Shashirekha, H. L. 2024. Hope on the horizon: Experiments with learning models for hope speech detection in spanish and english. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2024), co-located with the 40th Conference of the S...

    2024

  7. [15]

    M.; Nassif, A

    Elnagar, A.; Yagi, S. M.; Nassif, A. B.; Shahin, I.; and Salloum, S. A. 2021. Systematic literature review of dialectal Arabic: identification and detection. IEEE Access, 9: 31010--31042

    2021

  8. [16]

    D.-U.; Butler, S.; Ijomah, W.; Li, X.; and Zhou, H

    Fei, Z.; Yang, E.; Li, D. D.-U.; Butler, S.; Ijomah, W.; Li, X.; and Zhou, H. 2020. Deep convolution network based emotion analysis towards mental health care. Neurocomputing, 388: 212--227

    2020

  9. [17]

    D.; Shutova, E.; Lichtenstein, P.; De Melo, G.; and Gilardi, L

    Guti \'e rrez, E. D.; Shutova, E.; Lichtenstein, P.; De Melo, G.; and Gilardi, L. 2016. Detecting cross-cultural differences using a multilingual topic model. Transactions of the Association for Computational Linguistics, 4: 47--60

    2016

  10. [18]

    T.; Huang, Y.-H.; Wu, J.; and Chen, Y.-S

    Saravia, E.; Liu, H.-C. T.; Huang, Y.-H.; Wu, J.; and Chen, Y.-S. 2018. CARER : Contextualized Affect Representations for Emotion Recognition. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 3687--3697. Brussels, Belgium: Association ...

    2018

  11. [19]

    R.; and Wallbott, H

    Scherer, K. R.; and Wallbott, H. G. 1994. Evidence for universality and cultural variation of differential emotion response patterning. Journal of personality and social psychology, 66(2): 310

    1994

  12. [20]

    Shahriar, S.; Al Roken, N.; and Zualkernan, I. 2023. Classification of arabic poetry emotions using deep learning. Computers, 12(5): 89

    2023

  13. [21]

    Teahan, W. J. 2019. A new arabic dataset for emotion recognition. In Intelligent Computing: Proceedings of the 2019 Computing Conference, Volume 2, 200--216. Springer

    2019

  14. [22]

    Yang, J.; Huang, Q.; Ding, T.; Lischinski, D.; Cohen-Or, D.; and Huang, H. 2023. Emoset: A large-scale visual emotion dataset with rich attributes. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 20383--20394

    2023

  15. [23]

    Zaghouani, W.; Mubarak, H.; and Biswas, M. R. 2024. So hateful! Building a multi-label hate speech annotated Arabic dataset. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), 15044--15055

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.