REVIEW 3 major objections 4 minor 39 references
CEHA: A Dataset of Conflict Events in the Horn of Africa
T0 review · 3 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A 500-event benchmark maps the causes behind Horn of Africa conflict.
desk verdict Useful new conflict-event dataset for a low-resource region, with a genuine but addressable bias in the relevance test set from the GDELT pre-filter. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the CEHA dataset itself: 500 English event descriptions with gold labels. Its distinctive mechanism is the annotation scheme, which first separates relevant violent-conflict events from irrelevant posts using three criteria (Horn of Africa location, violence directed at people, and a specific rather than summary event), then assigns multi-label event types that emphasize the cause of the conflict rather than the action. The four cause-focused types are tribal/communal/ethnic conflict, religious conflict, socio-political violence against women, and climate-related security risks, with an Other category used only when none of the four apply. This scheme is what turns generic conflict event detection into a benchmark that can measure whether a model understands the conflict dynamics stakeholders care about.
What would settle it
Take a fresh random sample of GDELT conflict-coded Horn-of-Africa articles, have the same annotators label them without any model pre-filter, and compare the relevance distribution and model F1 against CEHA's test set; large differences would show the pre-filter biased the benchmark.
Extended reading notes
Core claim
On its own terms, the paper's discovery is CEHA itself and the evidence that the tasks it defines are non-trivial. Each of the 500 descriptions comes from ACLED or GDELT, 250 from each, and was labeled by subject matter experts working from a refined annotation guideline; 310 descriptions are conflict-relevant, and among relevant events the four event types are distributed unevenly, with tribal/communal/ethnic conflict most common at 115 instances and climate-related security risks rarest at 23. On the held-out test set of 250 items, the best supervised model reaches 83.09 F1 on relevance and 74.80 F1 on event-type classification, while the best six-shot LLM reaches 87.16 and 75.80 respectively, and most models show comparatively low precision on relevance. The authors read this as evidence that fine-grained conflict-event identification in the region remains an open challenge rather than a solved one.
Load-bearing premise
The sampling of irrelevant examples relies on an automatic language-model filter that removed only 38 percent of the irrelevant posts it should have removed, so the No examples in CEHA may not represent the mix of irrelevant content a live news feed would deliver.
Editorial extensions
If this is right
- Event-relevance and event-type classification in the Horn of Africa remain hard: the best relevance F1 is 87.16 and the best event-type F1 is 75.80.
- A small expert-annotated set of 500 items is enough to expose meaningful performance differences across models in low-resource settings.
- Few-shot LLM prompting is a viable route for low-resource conflict-event classification, often matching or surpassing fine-tuned BERT, RoBERTa, and T5 baselines.
- Climate-related security risks are the hardest event type to classify, partly because they are rare and under-represented in existing training resources.
- CEHA is positioned as a shared evaluation benchmark for future conflict-event and AI-for-social-good NLP work in the region.
Reading between the lines
- The automatic pre-filter used to clean GDELT posts achieved only 38 percent recall on the irrelevant class, so CEHA's relevance benchmark likely over-represents easy negative examples; a version built from an unfiltered random sample would test whether the relevance conclusions generalize.
- The cause-focused event-type scheme could be transferred to other conflict-prone regions, reusing the four definitions with local expert calibration.
- Because CEHA is English-only, its labels could seed cross-lingual annotation or evaluation of Amharic, Somali, and Arabic news sources.
- The low F1 for climate-related security risks points to a concrete gap: building larger targeted collections of climate-conflict events may improve model performance more than general-purpose scaling.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CEHA, a new benchmark dataset of 500 English-language event descriptions from ACLED and GDELT focusing on violent conflict events in the Horn of Africa. The dataset provides binary Event-relevance labels and fine-grained multi-label Event-type annotations (Tribal/Communal/Ethnic Conflict, Religious Conflict, Socio-political Violence Against Women, Climate-Related Security Risks) produced by subject-matter experts. The authors describe an iterative annotation-guideline development process, two pilot studies with inter-annotator agreement measurement, and a filtering step that used a few-shot Mistral-large model to remove irrelevant GDELT posts before human annotation. They then benchmark supervised models (BERT, RoBERTa, T5) and several LLMs (Mixtral, Mistral-large, DBRX, GPT-4o, Llama3) on both tasks in zero-shot, few-shot, and fine-tuned settings, reporting that the tasks are challenging in low-resource conditions and that LLMs benefit from in-context learning.
Significance. If the dataset is sound, it fills a real gap: existing conflict-event resources such as ACLED and GDELT do not provide fine-grained, region-specific event types that are directly relevant to humanitarian-peace-development stakeholders in the Horn of Africa. The paper's strengths include a transparent annotation process with domain experts, two pilot rounds with reported Cohen's kappa, clearly summarized label definitions, and a broadly scoped baseline comparison across multiple model families. The public release of the dataset and code is also a positive contribution. However, the reliability of the Event-relevance benchmark is undermined by the GDELT pre-filtering step, which selected test-set negatives based on the very model that is later evaluated on them. This issue is load-bearing for the central claim that the relevance task is challenging and that the dataset is a reusable evaluation resource.
major comments (3)
- [Section 3.3] The GDELT pre-filtering step is not distribution-neutral. The authors report that the Mistral-large filter achieved 38% recall on the No class (Appendix C), meaning it removed only the irrelevant posts it could confidently recognize and retained the harder ones, many of which were its own false negatives. Because the final test set is drawn from this filtered pool, the Event-relevance test distribution is shaped by Mistral-large's error pattern. In Section 5.2 and Table 8, Mistral-large is then evaluated on this exact filtered test set, so its F1 score and the observed difficulty ordering of models are not independent of the data-generation process. This is not circularity in label generation (the gold labels are human), but it is a selection bias that compromises the benchmark's validity for relevance classification. Please provide per-source and per-filter-status performance, or re-sample the test No examples from unfiltered GDELT and show that the main comparative conclusions still hold.
- [Section 3.3] The final annotation protocol assigns each item to a single annotator, with no adjudication of disagreements. The measured average pairwise Cohen's kappa for Event-relevance is only 0.63, which is moderate agreement; for a benchmark intended to evaluate models, label noise at this level can materially affect reported scores. The paper should either add an adjudication step for disagreements, report the distribution of labels per annotator in the test split, or explicitly quantify how single-annotator noise affects the reliability of the benchmark conclusions.
- [Section 5.2] The claim that 'precision remains relatively low for most LLM model variations, which demonstrates the challenging nature of the Event-relevance Classification task' is directly tied to the filtered test set. Since the filtered No examples are disproportionately those that Mistral-large failed to reject, the low precision observed for many models may be inflated by the sampling procedure rather than reflecting the true difficulty of relevance classification on raw GDELT data. The authors should temper this conclusion or provide evidence that the filtering bias does not drive the result.
minor comments (4)
- [Section 3.4] The paper describes the 4:1:5 split as providing a 'robust benchmark (test) set'; with 250 test examples and only 23 Climate-Related Security Risks events total, the per-class test sizes are small, and the term 'robust' is an overstatement.
- [Table 10] There is an inconsistent reference to 'GPT4' in the prose discussing Table 10, while the model is elsewhere called 'GPT-4o'; please harmonize the naming.
- [Section 3.2] The ACLED sample covers 2015-2024 while the GDELT sample covers only 2020-2024; this time-period mismatch should be stated as a potential source of distribution shift, and its possible effect on the benchmark results should be discussed.
- [Appendix C] The paper reports the filter's performance only on the No class; reporting precision, recall, and F1 for the Yes class as well would clarify how the filter's behavior differs between classes and help readers assess the selection bias more completely.
Circularity Check
No significant circularity: CEHA's gold labels are human-expert judgments and the evaluated models do not generate the ground truth; the Mistral-large prefilter is a sampling step, not a label source.
full rationale
The paper's central deliverable is a human-annotated benchmark. Section 3.3 states that each data point was annotated following a two-step process by subject matter experts, with Event-relevance and Event-type labels determined by annotators using definitions in Tables 2 and 3; the evaluated models in Section 5 are never used to produce gold labels. The only model-in-the-loop step is the GDELT sampling filter in Section 3.2, where a few-shot Mistral-large model was applied 'to remove irrelevant posts' before sampling. Appendix C reports its No-class recall as 38%, so the filter is selective rather than accurate. Because the filter's outputs are not treated as ground truth and the final relevance labels come from experts, evaluating Mistral-large (and other models) on the resulting test set is not a case of a fitted parameter being renamed as a prediction or of a claim being equivalent to its input by construction. The selection bias concern—that the filtered No distribution may not match real-world GDELT relevance distribution—is a validity and generalizability limitation, and the paper itself discloses the distributional mismatch in Limitations ('there are fewer samples for the "No" class for event-relevance in our dataset, which differs from the actual distribution in the real world'). This is a legitimate methodological concern, but it does not make the benchmark evaluation circular. The related-work citations to prior work by overlapping authors (Ran et al. 2023; Lu et al. 2023) are used for task formulation and contextualization, not as the load-bearing justification for CEHA's correctness or uniqueness. No uniqueness theorem, ansatz-by-citation, or definitional equivalence is invoked. Therefore the derivation chain is self-contained with respect to the expert annotations, and the circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption Expert annotations are treated as gold truth for relevance and event types.
- domain assumption The four event types (Tribal/Communal/Ethnic, Religious, Socio-political Violence Against Women, Climate-Related Security Risks) are the key fine-grained conflict categories for the Horn of Africa.
- ad hoc to paper Mistral-large pre-filtering does not materially bias the sampled GDELT irrelevant posts.
Cite this review
Pith. "Pith review of CEHA: A Dataset of Conflict Events in the Horn of Africa." pith.science (2026). https://pith.science/paper/B7O3HGNS
@misc{pith2026241213511,
author = {Pith},
title = {Pith review of: CEHA: A Dataset of Conflict Events in the Horn of Africa},
year = {2026},
howpublished = {\url{https://pith.science/paper/B7O3HGNS}},
note = {Machine review of arXiv:2412.13511}
}
read the original abstract
Natural Language Processing (NLP) of news articles can play an important role in understanding the dynamics and causes of violent conflict. Despite the availability of datasets categorizing various conflict events, the existing labels often do not cover all of the fine-grained violent conflict event types relevant to areas like the Horn of Africa. In this paper, we introduce a new benchmark dataset Conflict Events in the Horn of Africa region (CEHA) and propose a new task for identifying violent conflict events using online resources with this dataset. The dataset consists of 500 English event descriptions regarding conflict events in the Horn of Africa region with fine-grained event-type definitions that emphasize the cause of the conflict. This dataset categorizes the key types of conflict risk according to specific areas required by stakeholders in the Humanitarian-Peace-Development Nexus. Additionally, we conduct extensive experiments on two tasks supported by this dataset: Event-relevance Classification and Event-type Classification. Our baseline models demonstrate the challenging nature of these tasks and the usefulness of our dataset for model evaluations in low-resource settings with limited number of training data.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Meta AI. 2024 a . https://ai.meta.com/blog/meta-llama-3/ Introducing meta llama 3: The most capable openly available llm to date
work page 2024
-
[4]
Mistral AI. 2024 b . https://mistral.ai/news/mistral-large/ Au large
work page 2024
-
[5]
Sven Chojnacki, Christian Ickler, Michael Spies, and John Wiesel. 2012. Event data on armed conflict and security: New perspectives, old challenges, and some solutions. International Interactions, 38(4):382--401
work page 2012
-
[6]
Tirthankar Dasgupta, Abir Naskar, Rupsa Saha, and Lipika Dey. 2017. Crimeprofiler: Crime information extraction and visualization from news media. In Proceedings of the international conference on web intelligence, pages 541--549
work page 2017
-
[7]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805
arXiv 2018
-
[8]
Doddington, Alexis Mitchell, Mark A
George R. Doddington, Alexis Mitchell, Mark A. Przybocki, Lance A. Ramshaw, Stephanie Strassel, and Ralph M. Weischedel. 2004. https://api.semanticscholar.org/CorpusID:9776219 The automatic content extraction (ace) program – tasks, data, and evaluation . In International Conference on Language Resources and Evaluation
work page 2004
Show all 39 references
-
[9]
Dunford, Erin C
Karsten Donnay, Eric T. Dunford, Erin C. McGrath, David Backer, and David E. Cunningham. 2019. https://doi.org/10.1177/0022002718777050 Integrating conflict event data . Journal of Conflict Resolution, 63(5):1337--1364
2019 doi
-
[10]
Deborah J Gerner, Philip A Schrodt, Omur Yilmaz, and Rajaa Abu-Jabr. 2002. The creation of cameo (conflict and mediation event observations): An event data framework for a post cold war world. In annual meeting of the American Political Science Association, volume 29
2002
-
[11]
Andrew Halterman, Benjamin E Bagozzi, Andreas Beger, Phil Schrodt, and Grace Scraborough. 2023. Plover and polecat: A new political event ontology and dataset. In International Studies Association Conference Paper
2023
-
[12]
Bekele Abera Hordofa. 2020. Event extraction and representation model from news articles. International Journal of Innovations in Engineering and Technology, 16(3):1--8
2020
-
[13]
Tobias Ide, Michael Brzoska, Jonathan F Donges, and Carl-Friedrich Schleussner. 2020. Multi-method evidence for when and how climate-related disasters contribute to armed conflict risk. Global Environmental Change, 62:102063
2020
-
[14]
Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088
2024 arXiv
-
[15]
Brenden Jongman, Jurjen Wagemaker, Beatriz Revilla Romero, and Erin Coughlan de Perez. 2015. Early flood detection for rapid humanitarian response: harnessing near real-time satellite and twitter signals. ISPRS International Journal of Geo-Information, 4(4):2246--2266
2015
-
[16]
Shaheen Khatoon, Majed A Alshamari, Amna Asif, Md Maruf Hasan, Sherif Abdou, Khaled Mostafa Elsayed, and Mohsen Rashwan. 2021. Development of social media analytics system for emergency event detection and crisismanagement. Comput. Mater. Contin, 68(3)
2021
-
[17]
Jacob Kurtzer, Sierra Ballard, and Hareem Fatima Abdullah. 2022. https://www.csis.org/analysis/concurrent-crises-horn-africa Concurrent Crises in the Horn of Africa
2022
-
[18]
Gary LaFree and Laura Dugan. 2007. Introducing the global terrorism database. Terrorism and political violence, 19(2):181--204
2007
-
[19]
u reder, Barbara Riedler, Lorenz Wendt, Andreas Braun, Dirk Tiede, Elisabeth Schoepfer, Peter Zeil, Kristin Spr \
Stefan Lang, Petra F \"u reder, Barbara Riedler, Lorenz Wendt, Andreas Braun, Dirk Tiede, Elisabeth Schoepfer, Peter Zeil, Kristin Spr \"o hnle, Kerstin Kulessa, et al. 2020. Earth observation tools and services to increase the effectiveness of humanitarian assistance. Europea...
2020
-
[20]
Kalev Leetaru and Philip A Schrodt. 2013. Gdelt: Global data on events, location, and tone, 1979--2012. In ISA annual convention, volume 2, pages 1--49. Citeseer
2013
-
[21]
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692
2019 arXiv
-
[22]
Di Lu, Shihao Ran, Joel Tetreault, and Alejandro Jaimes. 2023. https://doi.org/10.18653/v1/2023.acl-short.143 Event extraction as question generation and answering . In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Pap...
2023 doi
-
[23]
Muhabie Mekonnen Mengistu. 2015. The root causes of conflicts in the horn of africa. American Journal of Applied Psychology, 4(2):28--34
2015
-
[24]
Tim Nugent, Fabio Petroni, Natraj Raman, Lucas Carstens, and Jochen L Leidner. 2017. A comparison of classification models for natural disaster and critical event detection from news. In 2017 IEEE international conference on big data (Big Data), pages 3750--3759. IEEE
2017
-
[25]
OCHA . 2024. https://www.unocha.org/publications/report/ethiopia/greater-horn-africa-humanitarian-key-messages-february-2024 Greater horn of africa humanitarian key messages, february 2024 . Accessed: 2024-06-15
2024
-
[26]
Helene Olsen, \'E tienne Simon, Erik Velldal, and Lilja vrelid. 2024. https://aclanthology.org/2024.case-1.5 Socio-political events of conflict and unrest: A survey of available datasets . In Proceedings of the 7th Workshop on Challenges and Applications of Automated Extractio...
2024
-
[27]
OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774
2023 arXiv
-
[28]
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2023. https://arxiv.org/abs/1910.10683 Exploring the limits of transfer learning with a unified text-to-text transformer . Preprint, arXiv:1910.10683
2023 arXiv
-
[29]
Clionadh Raleigh, Roudabeh Kishi, and Andrew Linke. 2023. https://doi.org/10.1057/s41599-023-01559-4 Political instability patterns are obscured by conflict dataset scope conditions, sources, and coding choices . Humanities and Social Sciences Communications, 10:74
2023 doi
-
[30]
Shihao Ran, Di Lu, Joel Tetreault, Aoife Cahill, and Alejandro Jaimes. 2023. A new task and dataset on detecting attacks on human rights defenders. arXiv preprint arXiv:2306.17695
2023 arXiv
-
[31]
Idean Salehyan, Cullen S Hendrix, Jesse Hamner, Christina Case, Christopher Linebarger, Emily Stull, and Jennifer Williams. 2012. Social conflict in africa: A new database. International Interactions, 38(4):503--511
2012
-
[32]
Philip A Schrodt. 2012. Cameo: Conflict and mediation event observations event and actor codebook. Pennsylvania State University, 610:35
2012
-
[33]
Andrew Shaver, Hannah Kazis-Taylor, Claudia Loomis, Mia Bartschi, Paul Patterson, Adrian Vera, Kevin Abad, Saher Alqarwani, Clay Bell, Sebastian Bock, Kieran Cabezas, Heidi Felix, Jennifer Gonzalez, Christopher Hoeft, Aileen Martinez, Kai Keltner, Jessica Moroyoqui, Kieko Pama...
2023
-
[34]
Sonit Singh. 2018. Natural language processing for information extraction. arXiv preprint arXiv:1807.02383
2018 arXiv
-
[35]
Negasi Solomon, Emiru Birhane, Christopher Gordon, Mebrahtu Haile, Fatemeh Taheri, Hossein Azadi, and J \"u rgen Scheffran. 2018. Environmental impacts and causes of conflict in the horn of africa: A review. Earth-science reviews, 177:284--290
2018
-
[36]
Zhiyi Song, Ann Bies, Stephanie Strassel, Tom Riese, Justin Mott, Joe Ellis, Jonathan Wright, Seth Kulick, Neville Ryant, and Xiaoyi Ma. 2015. From light to rich ere: Annotation of entities, relations, and events. In Proceedings of the the 3rd Workshop on EVENTS: Definition, D...
2015
-
[37]
Ralph Sundberg and Erik Melander. 2013. https://doi.org/10.1177/0022343313484347 Introducing the ucdp georeferenced event dataset . Journal of Peace Research, 50(4):523--532
2013 doi
-
[38]
The Mosaic Research Team. 2024. https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm Introducing dbrx: A new state-of-the-art open llm
2024
-
[39]
Wei Wang. 2018. Event detection and extraction from news articles. Ph.D. thesis, Virginia Tech
2018
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.