GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP

Abdul Waheed; El Moatez Billah Nagoudi; Md Tawkat Islam Khondaker; Muhammad Abdul-Mageed

GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2305.14976 v2 pith:4WCDDSFR submitted 2023-05-24 cs.CL cs.LG

GPTAraEval: A Comprehensive Evaluation of ChatGPT on Arabic NLP

Md Tawkat Islam Khondaker , Abdul Waheed , El Moatez Billah Nagoudi , Muhammad Abdul-Mageed This is my paper

classification cs.CL cs.LG

keywords chatgptarabicevaluationperformancecomprehensivedialectalenglishfurther

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

0 comments

read the original abstract

ChatGPT's emergence heralds a transformative phase in NLP, particularly demonstrated through its excellent performance on many English benchmarks. However, the model's efficacy across diverse linguistic contexts remains largely uncharted territory. This work aims to bridge this knowledge gap, with a primary focus on assessing ChatGPT's capabilities on Arabic languages and dialectal varieties. Our comprehensive study conducts a large-scale automated and human evaluation of ChatGPT, encompassing 44 distinct language understanding and generation tasks on over 60 different datasets. To our knowledge, this marks the first extensive performance analysis of ChatGPT's deployment in Arabic NLP. Our findings indicate that, despite its remarkable performance in English, ChatGPT is consistently surpassed by smaller models that have undergone finetuning on Arabic. We further undertake a meticulous comparison of ChatGPT and GPT-4's Modern Standard Arabic (MSA) and Dialectal Arabic (DA), unveiling the relative shortcomings of both models in handling Arabic dialects compared to MSA. Although we further explore and confirm the utility of employing GPT-4 as a potential alternative for human evaluation, our work adds to a growing body of research underscoring the limitations of ChatGPT.

discussion (0)

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

Evaluation of Adversarial Robustness in Arabic Language Models
cs.CL 2026-07 conditional novelty 5.0

Arabic BERT-family sentiment models lose up to 92% accuracy under diacritics and 58% under conjunction attacks; paraphrase attacks cut accuracy by 76% on average, and adversarial training only partially helps.