pith. machine review for the scientific record. sign in

arxiv: 2604.08566 · v1 · submitted 2026-03-18 · 💻 cs.CL · cs.LG

Recognition: no theorem link

Sentiment Classification of Gaza War Headlines: A Comparative Analysis of Large Language Models and Arabic Fine-Tuned BERT Models

Amr Eleraqi , Hager H. Mustafa , Abdul Hadi N. Ahmed
Authors on Pith no claims yet

Pith reviewed 2026-05-15 09:31 UTC · model grok-4.3

classification 💻 cs.CL cs.LG
keywords sentiment classificationlarge language modelsArabic BERTGaza Warmedia framingconflict narrativesalgorithmic interpretationdistributional metrics
0
0 comments X

The pith

Different AI models apply distinct interpretive lenses when classifying sentiment in Gaza War headlines.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper compares three large language models and six fine-tuned Arabic BERT models on sentiment classification for 10,990 Arabic headlines about the 2023 Gaza War. Rather than measuring accuracy against human labels, it treats model outputs as interpretive acts and quantifies their systematic differences with entropy, distance, and variance metrics. BERT models tend toward neutral labels while LLMs, especially LLaMA-3.1-8B, push strongly negative; GPT-4.1 alone adjusts outputs according to narrative frames such as humanitarian or security. A reader would care because the results show that automated sentiment scores are not neutral measurements but active framings of conflict media. If the claim holds, studies that rely on any single model's output for media tone in war contexts rest on an unexamined choice of lens.

Core claim

Sentiment classification of conflict-related media is an interpretive act produced by model architecture. On a corpus of 10,990 Arabic news headlines, fine-tuned BERT models exhibit a strong bias toward neutral classifications while LLMs consistently amplify negative sentiment, with LLaMA-3.1-8B showing near-total collapse into negativity. Frame-conditioned analysis shows that GPT-4.1 modulates its judgments in line with narrative frames whereas other LLMs display limited contextual modulation. The choice of model therefore constitutes a choice of interpretive lens that shapes how conflict narratives are algorithmically framed and emotionally evaluated.

What carries the argument

Comparative distributional analysis that measures divergence across models via Shannon Entropy, Jensen-Shannon Distance, and a Variance Score of deviation from aggregate behavior.

If this is right

  • Automated sentiment tools cannot be treated as interchangeable measures of media tone in war reporting.
  • Studies in computational social science that use single-model sentiment outputs risk embedding one architecture's framing as neutral fact.
  • Frame sensitivity is model-dependent, appearing reliably only in certain LLMs such as GPT-4.1.
  • Epistemological approaches that foreground algorithmic discrepancy become necessary when applying these tools to conflict discourse.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Media analysts may need to report sentiment ranges across multiple model families rather than single scores.
  • The same approach could be applied to other polarized topics to test whether interpretive divergence is a general feature of current AI sentiment systems.
  • Prompt engineering or ensemble voting might reduce but not eliminate the observed architectural differences.
  • Without a human gold standard the study leaves open which lens, if any, aligns with public perception of the headlines.

Load-bearing premise

Observed differences in model outputs reflect genuine interpretive differences rather than artifacts of training data, prompting, or fine-tuning choices, even without any human-annotated gold standard.

What would settle it

A human-annotated gold standard for the same headlines that shows all models converging on similar sentiment distributions or matching the human labels would falsify the claim that divergences represent distinct interpretive lenses.

Figures

Figures reproduced from arXiv: 2604.08566 by Abdul Hadi N. Ahmed, Amr Eleraqi, Hager H. Mustafa.

Figure 1
Figure 1. Figure 1: Distribution of Headlines Across the Five Key Terms [PITH_FULL_IMAGE:figures/full_fig_p012_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of Headline Word Counts Source: Author’s calculations based on the study dataset. Finally, headline lengths are broadly comparable across keyword subsets. Mean cleaned length ranges narrowly from 8.6 to 9.5 words (Gaza: 9.5; al-Qassam Brigades: 9.2; Israeli army: 9.0; Hamas: 8.7; Captives: 8.6), and the following table summarizes show similar medians and dispersion across groups with occasiona… view at source ↗
read the original abstract

This study examines how different artificial intelligence architectures interpret sentiment in conflict-related media discourse, using the 2023 Gaza War as a case study. Drawing on a corpus of 10,990 Arabic news headlines (Eleraqi 2026), the research conducts a comparative analysis between three large language models and six fine-tuned Arabic BERT models. Rather than evaluating accuracy against a single human-annotated gold standard, the study adopts an epistemological approach that treats sentiment classification as an interpretive act produced by model architectures. To quantify systematic differences across models, the analysis employs information-theoretic and distributional metrics, including Shannon Entropy, Jensen-Shannon Distance, and a Variance Score measuring deviation from aggregate model behavior. The results reveal pronounced and non-random divergence in sentiment distributions. Fine-tuned BERT models, particularly MARBERT, exhibit a strong bias toward neutral classifications, while LLMs consistently amplify negative sentiment, with LLaMA-3.1-8B showing near-total collapse into negativity. Frame-conditioned analysis further demonstrates that GPT-4.1 adjusts sentiment judgments in line with narrative frames (e.g., humanitarian, legal, security), whereas other LLMs display limited contextual modulation. These findings suggest that the choice of model constitutes a choice of interpretive lens, shaping how conflict narratives are algorithmically framed and emotionally evaluated. The study contributes to media studies and computational social science by foregrounding algorithmic discrepancy as an object of analysis and by highlighting the risks of treating automated sentiment outputs as neutral or interchangeable measures of media tone in contexts of war and crisis.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper claims that different AI models apply distinct interpretive lenses to sentiment in 10,990 Arabic Gaza War headlines, with fine-tuned BERT models (especially MARBERT) showing strong neutral bias and LLMs (especially LLaMA-3.1-8B) showing pronounced negative bias. Using Shannon entropy, Jensen-Shannon distance, and variance-from-aggregate metrics, it argues that model choice shapes algorithmic framing of conflict narratives and that these divergences are systematic rather than random.

Significance. If the reported distributional divergences prove robust, the work usefully foregrounds algorithmic discrepancy as an object of study in computational social science and media studies. The information-theoretic metrics provide a reproducible way to quantify model disagreement on sensitive topics, and the frame-conditioned analysis of GPT-4.1 offers a concrete illustration of contextual modulation.

major comments (2)
  1. [Abstract] Abstract and epistemological framing section: the central claim that observed divergences demonstrate genuine interpretive differences (rather than artifacts of pretraining, fine-tuning, or prompting) is load-bearing yet unsupported by any human-annotated reference labels. The paper explicitly forgoes gold-standard validation, leaving the interpretation of LLaMA collapse versus MARBERT neutrality open to alternative explanations.
  2. [Methodology] Methodology and results sections: no implementation details, hyperparameter settings, prompt templates, or statistical significance tests are supplied for the entropy, Jensen-Shannon, or variance metrics. Without these, it is impossible to verify the reported patterns (e.g., near-total negativity in LLaMA-3.1 or frame sensitivity in GPT-4.1) or to rule out prompt-induced artifacts.
minor comments (1)
  1. [Abstract] Corpus citation: the source is listed as Eleraqi 2026 by the lead author; a brief statement on data-construction independence or selection criteria would strengthen transparency.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments. We address the major points below and revise the manuscript accordingly to strengthen transparency while preserving the paper's epistemological framing.

read point-by-point responses

  1. Referee: [Abstract] Abstract and epistemological framing section: the central claim that observed divergences demonstrate genuine interpretive differences (rather than artifacts of pretraining, fine-tuning, or prompting) is load-bearing yet unsupported by any human-annotated reference labels. The paper explicitly forgoes gold-standard validation, leaving the interpretation of LLaMA collapse versus MARBERT neutrality open to alternative explanations.

    Authors: Our study deliberately adopts an epistemological approach that treats sentiment classification as an interpretive act produced by model architectures, rather than seeking validation against a single human gold standard. The central claim concerns the existence of systematic, non-random divergences (quantified via Shannon entropy, Jensen-Shannon distance, and variance-from-aggregate on the full 10,990 headlines), not which model is objectively correct. We acknowledge that pretraining biases and prompting effects remain possible alternative explanations. In revision we will expand the framing section to state this explicitly and add a limitations paragraph discussing these alternatives without claiming the divergences prove superior accuracy. revision: partial

  2. Referee: [Methodology] Methodology and results sections: no implementation details, hyperparameter settings, prompt templates, or statistical significance tests are supplied for the entropy, Jensen-Shannon, or variance metrics. Without these, it is impossible to verify the reported patterns (e.g., near-total negativity in LLaMA-3.1 or frame sensitivity in GPT-4.1) or to rule out prompt-induced artifacts.

    Authors: We agree that the submitted manuscript omitted necessary implementation details. The revised version will supply: full prompt templates for all LLMs (including frame-conditioned variants), hyperparameter settings for the six fine-tuned Arabic BERT models, exact formulas and computation procedures for Shannon entropy, Jensen-Shannon distance, and the variance score, and statistical significance tests (permutation tests and bootstrap confidence intervals) confirming the non-random character of the divergences. These additions will support reproducibility and allow readers to assess potential prompt artifacts. revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper draws on a self-cited corpus (Eleraqi 2026) as input data and applies standard, externally defined metrics (Shannon Entropy, Jensen-Shannon Distance, Variance Score) to quantify divergences in model outputs. No equations or steps reduce a claimed prediction or result to the inputs by construction; the central claim that model choice constitutes an interpretive lens is presented as an epistemological interpretation of observed distributions rather than a mathematical derivation. The absence of a human gold standard is explicitly acknowledged as a deliberate framing choice and does not create a self-referential loop. The analysis remains self-contained with independent content from the comparative metrics and does not rely on load-bearing self-citations, smuggled ansatzes, or renamed known results.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on treating model outputs as interpretive acts without external validation and on a self-authored headline corpus; no free parameters or invented entities are introduced.

axioms (1)
  • domain assumption Sentiment classification by models is best treated as an interpretive act rather than an objective property requiring human gold-standard validation
    Explicitly adopted in the abstract as the epistemological approach.

pith-pipeline@v0.9.0 · 5588 in / 1257 out tokens · 57457 ms · 2026-05-15T09:31:19.602541+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 27 canonical work pages · 4 internal anchors

  1. [1]

    ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic

    “ARBERT & MARBERT: Deep Bidirectional Transformers for Arabic.” Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL 2021), Long Papers, 7088–7105. Association for Computational Linguistics. https://aclanthology.org/2021.acl-long.551.pdf Abuasaker, Walaa, Mónica Sánchez, Jennifer Nguyen, Nil Agell, Núria Agell, and ...

    work page 2021

  2. [2]

    Almutrash, Salman, and Shadi Abudalfa

    https://doi.org/10.3390/make7010008. Almutrash, Salman, and Shadi Abudalfa

    work page doi:10.3390/make7010008

  3. [3]

    Comparative Study on the Efficiency of Using PaLM and CAMeLBERT for Arabic Entity Sentiment Classification

    “Comparative Study on the Efficiency of Using PaLM and CAMeLBERT for Arabic Entity Sentiment Classification.” In SaudiCIS 2024 Proceedings (1st Saudi Conference on Information Systems, Dhahran, Saudi Arabia, November 19–21, 2024). AIS eLibrary. https://aisel.aisnet.org/saudicis2024/66 Antoun, Wissam, Fady Baly, and Hazem Hajj

    work page 2024

  4. [4]

    AraBERT: Transformer-based Model for Arabic Language Understanding

    “AraBERT: Transformer-based Model for Arabic Language Understanding.” In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection (OSACT), 9–15. Marseille, France: European Language Resource Association. https://aclanthology.org/2020.osact-1.2/. Bommasani, Rishi, et al

    work page 2020

  5. [5]

    On the Opportunities and Risks of Foundation Models

    “On the Opportunities and Risks of Foundation Models.” arXiv (August 2021). https://doi.org/10.48550/arXiv.2108.07258 41 Boudad, Naima, Rdouan Faizi, Rachid Oulad Haj Thami, and Raddouane Chiheb

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2108.07258 2021

  6. [6]

    Sentiment Analysis in Arabic: A Review of the Literature

    “Sentiment Analysis in Arabic: A Review of the Literature.” Ain Shams Engineering Journal 9 (4): 2479–2490. https://doi.org/10.1016/j.asej.2017.04.007. Ceron, Andrea, Luigi Curini, and Stefano M. Iacus

    work page doi:10.1016/j.asej.2017.04.007 2017

  7. [7]

    Using Sentiment Analysis to Monitor Electoral Campaigns: Method Matters—Evidence From the United States and Italy

    “Using Sentiment Analysis to Monitor Electoral Campaigns: Method Matters—Evidence From the United States and Italy.” Social Science Computer Review 33 (1): 3–20. https://doi.org/10.1177/0894439314521983. Devlin, Jacob, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova

    work page doi:10.1177/0894439314521983

  8. [8]

    doi:10.18653/v1/N19-1423 , pages =

    “BERT: Pre- training of Deep Bidirectional Transformers for Language Understanding.” In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 4171–4186. Minneapolis, Minnesota: Association for Computational Linguistics. https://doi.o...

    work page doi:10.18653/v1/n19-1423 2019

  9. [9]

    Arabic News Corpus on the Gaza War and Geopolitical Narratives (2023–2025)

    “Arabic News Corpus on the Gaza War and Geopolitical Narratives (2023–2025).” Harvard Dataverse, V1.0 (January 4, 2026). https://doi.org/10.7910/DVN/FFENX3. Entman, Robert M

    work page doi:10.7910/dvn/ffenx3 2023

  10. [10]

    Framing: Toward Clarification of a Fractured Paradigm

    “Framing: Toward Clarification of a Fractured Paradigm.” Journal of Communication 43 (4): 51–58. https://doi.org/10.1111/j.1460-2466.1993.tb01304.x Grimmer, Justin, and Brandon M. Stewart

    work page doi:10.1111/j.1460-2466.1993.tb01304.x 1993

  11. [11]

    Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts

    “Text as Data: The Promise and Pitfalls of Automatic Content Analysis Methods for Political Texts.” Political Analysis 21 (3): 267–297. https://doi.org/10.1093/pan/mps028. Gururangan, Suchin, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith

    work page doi:10.1093/pan/mps028

  12. [12]

    Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks

    “Don’t Stop Pretraining: Adapt Language Models to Domains and Tasks.” In Proceedings of the 58th Annual Meeting of the Association for 42 Computational Linguistics, 8342–8360. Online: Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-main.740. Habash, Nizar Y

    work page doi:10.18653/v1/2020.acl-main.740 2020

  13. [13]

    https://doi.org/10.2200/S00277ED1V01Y201008HLT010

    San Rafael, CA: Morgan & Claypool Publishers. https://doi.org/10.2200/S00277ED1V01Y201008HLT010. Hannani, Mohamed, Abdelhadi Soudi, and Kristof Van Laerhoven

    work page doi:10.2200/s00277ed1v01y201008hlt010

  14. [14]

    Assessing the Performance of ChatGPT-4, Fine-tuned BERT and Traditional ML Models on Moroccan Arabic Sentiment Analysis

    “Assessing the Performance of ChatGPT-4, Fine-tuned BERT and Traditional ML Models on Moroccan Arabic Sentiment Analysis.” In Proceedings of the 4th International Conference on Natural Language Processing for Digital Humanities (NLP4DH 2024). https://aclanthology.org/2024.nlp4dh-1.47.pdf. Haselmayer, Martin, and Marcelo Jenny

    work page 2024

  15. [15]

    Sentiment Analysis of Political Communication: Combining a Dictionary Approach with Crowdcoding

    “Sentiment Analysis of Political Communication: Combining a Dictionary Approach with Crowdcoding.” Quality & Quantity 51 (6): 2623–2646. https://doi.org/10.1007/s11135-016-0412-4. Huang, Lei, et al

    work page doi:10.1007/s11135-016-0412-4

  16. [16]

    A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions

    “A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions.” arXiv (November 2023). https://doi.org/10.48550/arXiv.2311.05232. Inoue, Go, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2311.05232 2023

  17. [17]

    The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models

    “The Interplay of Variant, Size, and Task Type in Arabic Pre-trained Language Models.” In Proceedings of the Sixth Arabic Natural Language Processing Workshop, 92–104. Kyiv, Ukraine (Virtual): Association for Computational Linguistics. https://aclanthology.org/2021.wanlp-1.10/. Ke, Zixuan, Yijia Shao, Haowei Lin, Hu Xu, Lei Shu, and Bing Liu

    work page 2021

  18. [18]

    Adapting a Language Model While Preserving its General Knowledge

    “Adapting a Language Model While Preserving its General Knowledge.” In Proceedings of the 2022 43 Conference on Empirical Methods in Natural Language Processing, 10177–10188. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics. https://doi.org/10.18653/v1/2022.emnlp-main.693. Kim, Yoon

    work page doi:10.18653/v1/2022.emnlp-main.693 2022

  19. [19]

    Convolutional Neural Networks for Sentence Classification

    “Convolutional Neural Networks for Sentence Classification.” In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), 1746–1751. Doha, Qatar: Association for Computational Linguistics. https://doi.org/10.3115/v1/D14-1181. Krippendorff, Klaus

    work page doi:10.3115/v1/d14-1181 2014

  20. [20]

    Content Analysis: An Introduction to Its Methodology. 4th ed. Thousand Oaks, CA: SAGE Publications, Inc. https://doi.org/10.4135/9781071878781. Kullback, Solomon, and Richard A. Leibler

    work page doi:10.4135/9781071878781

  21. [21]

    Chernoff

    “On Information and Sufficiency.” The Annals of Mathematical Statistics 22 (1): 79–86. https://doi.org/10.1214/aoms/1177729694. Liu, Pengfei, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig

    work page doi:10.1214/aoms/1177729694

  22. [22]

    Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

    “Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.” arXiv (July 2021). https://doi.org/10.48550/arXiv.2107.13586. McCombs, Maxwell E., and Donald L. Shaw

    work page doi:10.48550/arxiv.2107.13586 2021

  23. [23]

    The Agenda-Setting Function of Mass Media

    “The Agenda-Setting Function of Mass Media.” Public Opinion Quarterly 36 (2): 176–187. https://doi.org/10.1086/267990. Mikolov, Tomas, Kai Chen, Greg Corrado, and Jeffrey Dean

    work page doi:10.1086/267990

  24. [24]

    Efficient Estimation of Word Representations in Vector Space

    “Efficient Estimation of Word Representations in Vector Space.” arXiv:1301.3781. https://doi.org/10.48550/arXiv.1301.3781. 44 Mulki, Hala, Hatem Haddad, and Ismail Babaoğlu

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.1301.3781

  25. [25]

    Modern Trends in Arabic Sentiment Analysis: A Survey

    “Modern Trends in Arabic Sentiment Analysis: A Survey.” Traitement Automatique des Langues 58 (3): 15–39. https://aclanthology.org/2017.tal-3.3/ OpenAI

    work page 2017

  26. [26]

    GPT-4 Technical Report

    “GPT-4 Technical Report.” arXiv 2303.08774. https://doi.org/10.48550/arXiv.2303.08774. Shannon, Claude E

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2303.08774

  27. [27]

    A Mathematical Theory of Communication

    “A Mathematical Theory of Communication.” Bell System Technical Journal 27 (3): 379–423; 27 (4): 623–656. https://doi.org/10.1002/j.1538- 7305.1948.tb01338.x

    work page doi:10.1002/j.1538- 1948