Pith. sign in

REVIEW 50 references

Lemmatization as a Classification Task: Results from Arabic across Multiple Genres

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2506.18399 v1 pith:K2GDOKDV submitted 2025-06-23 cs.CL

Mostafa Saeed , Nizar Habash This is my paper
classification cs.CL
keywords lemmatizationarabicclassificationclusteringexistinggenreslimitedresults
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Lemmatization is crucial for NLP tasks in morphologically rich languages with ambiguous orthography like Arabic, but existing tools face challenges due to inconsistent standards and limited genre coverage. This paper introduces two novel approaches that frame lemmatization as classification into a Lemma-POS-Gloss (LPG) tagset, leveraging machine translation and semantic clustering. We also present a new Arabic lemmatization test set covering diverse genres, standardized alongside existing datasets. We evaluate character level sequence-to-sequence models, which perform competitively and offer complementary value, but are limited to lemma prediction (not LPG) and prone to hallucinating implausible forms. Our results show that classification and clustering yield more robust, interpretable outputs, setting new benchmarks for Arabic lemmatization.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

50 extracted references · 40 canonical work pages

  1. [1]

    Eman Abdelrahman, Fatimah Alotaibi, Edward A Fox, and Osman Balci. 2021. Otrouha: A corpus of arabic etds and a framework for automatic subject classification. The Journal of Electronic Theses and Dissertations, 1(1):6

    2021

  2. [2]

    Muhamed Al Khalil, Hind Saddiki, Nizar Habash, and Latifa Alfalasi. 2018. A Leveled Reading Corpus of Modern Standard A rabic . In Proceedings of the Language Resources and Evaluation Conference (LREC), Miyazaki, Japan

    2018

  3. [3]

    Riadh Belkebir and Nizar Habash. 2021. https://doi.org/10.18653/v1/2021.conll-1.47 Automatic error type annotation for A rabic . In Proceedings of the 25th Conference on Computational Natural Language Learning, pages 596--606, Online. Association for Computational Linguistics

    work page doi:10.18653/v1/2021.conll-1.47 2021

  4. [4]

    Toms Bergmanis and Sharon Goldwater. 2018 a . Context sensitive neural lemmatization with lematus. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 1391--1400

    2018

  5. [5]

    Toms Bergmanis and Sharon Goldwater. 2018 b . Context sensitive neural lemmatization with lematus. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1391--1400

    2018

  6. [6]

    Mohamed Boudchiche, Azzeddine Mazroui, Mohamed Ould Abdallahi Ould Bebah, Abdelhak Lakhouaja, and Abderrahim Boudlal. 2017. Alkhalil morpho sys 2: A robust arabic morpho-syntactic analyzer. Journal of King Saud University-Computer and Information Sciences, 29(2):141--146

    2017

  7. [7]

    Tim Buckwalter. 2002. Buckwalter A rabic morphological analyzer version 1.0. Linguistic Data Consortium (LDC) catalog number LDC2002L49, ISBN 1-58563-257-0

    2002

  8. [8]

    Costanza Conforti, Matthias Huck, and Alexander Fraser. 2018. Neural morphological tagging of lemma sequences for machine translation. In Proceedings of the 13th Conference of the Association for Machine Translation in the Americas (Volume 1: Research Track), pages 39--53

    2018

Show all 50 references
  1. [9]

    Mona Diab, Nizar Habash, Owen Rambow, and Ryan Roth. 2013. LDC A rabic treebanks and associated corpora: Data divisions manual . arXiv preprint arXiv:1309.5652

    2013 arXiv

  2. [10]

    Kais Dukes and Nizar Habash. 2010 a . Morphological annotation of quranic arabic. In Lrec, pages 2530--2536

    2010

  3. [11]

    Kais Dukes and Nizar Habash. 2010 b . Morphological Annotation of Quranic A rabic . In Proceedings of the Language Resources and Evaluation Conference (LREC), Valetta, Malta

    2010

  4. [12]

    Tarek El-Shishtawy and Fatma El-Ghannam. 2014. A lemma based evaluator for semitic language text summarization systems. arXiv preprint arXiv:1403.5596

    2014 arXiv

  5. [13]

    Salman Elgamal, Ossama Obeid, Tameem Kabbani, Go Inoue, and Nizar Habash. 2024. Arabic diacritics in the wild: Exploiting opportunities for improved diacritization. arXiv preprint arXiv:2406.05760

    2024 arXiv

  6. [14]

    Andrea Gesmundo and Tanja Samard z i \'c . 2012. https://aclanthology.org/P12-2072 Lemmatisation as a tagging task . In Proceedings of the 50th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 368--372, Jeju Island, Korea. Associa...

    2012

  7. [15]

    David Graff, Mohamed Maamouri, Basma Bouziri, Sondos Krouna, Seth Kulick, and Tim Buckwalter. 2009. Standard A rabic Morphological Analyzer (SAMA) Version 3.1 . Linguistic Data Consortium LDC2009E73

    2009

  8. [16]

    Nizar Habash and David Palfreyman. 2022. https://aclanthology.org/2022.lrec-1.9 ZAEBUC : An annotated A rabic- E nglish bilingual writer corpus . In Proceedings of the Thirteenth Language Resources and Evaluation Conference, pages 79--88, Marseille, France. European Language R...

    2022

  9. [17]

    Nizar Habash, Hanada Taha-Thomure, Khalid N Elmadani, Zeina Zeino, and Abdallah Abushmaes. 2024. Guidelines for fine-grained sentence-level arabic readability annotation. arXiv preprint arXiv:2410.08674

    2024 arXiv

  10. [18]

    Anton Karl Ingason, Sigr \'u n Helgad \'o ttir, Hrafn Loftsson, and Eir \' kur R \"o gnvaldsson. 2008. A mixed method lemmatization algorithm using a hierarchy of linguistic identities (holi). In Advances in natural language processing: 6th international conference, GoTAL 2008...

    2008

  11. [19]

    Svanhv \' t Lilja Ing \'o lfsd \'o ttir, Hrafn Loftsson, J \'o n Fri rik Da ason, and Krist \' n Bjarnad \'o ttir. 2019. Nefnir: A high accuracy lemmatizer for icelandic. arXiv preprint arXiv:1907.11907

    2019 arXiv

  12. [20]

    Go Inoue, Bashar Alhafni, Nurpeiis Baimukan, Houda Bouamor, and Nizar Habash. 2021 a . The interplay of variant, size, and task type in arabic pre-trained language models. arXiv preprint arXiv:2103.06678

    2021 arXiv

  13. [21]

    Go Inoue, Salam Khalifa, and Nizar Habash. 2021 b . Morphosyntactic tagging with pre-trained language models for arabic and its dialects. arXiv preprint arXiv:2110.06852

    2021 arXiv

  14. [22]

    Mustafa Jarrar, Diyam Akra, and Tymaa Hammouda. 2024. Alma: Fast lemmatizer and pos tagger for arabic. Procedia Computer Science, 244:378--387

    2024

  15. [23]

    Bart Jongejan and Hercules Dalianis. 2009. Automatic training of lemmatization rules that handle morphological changes in pre-, in-and suffixes alike. In Proceedings of the Joint Conference of the 47th Annual Meeting of the ACL and the 4th International Joint Conference on Nat...

    2009

  16. [24]

    Christian Khairallah, Salam Khalifa, Reham Marzouk, Mayar Nassar, and Nizar Habash. 2024. Camel morph msa: A large-scale open-source morphological analyzer for modern standard arabic. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Langu...

    2024

  17. [25]

    Salam Khalifa, Nasser Zalmout, and Nizar Habash. 2020. Morphological analysis and disambiguation for gulf arabic: The interplay between resources and methods. In Proceedings of the Twelfth Language Resources and Evaluation Conference, pages 3895--3904

    2020

  18. [26]

    Daniel Kondratyuk, Tom \'a s Gaven c iak, Milan Straka, and Jan Haji c . 2018 a . Lemmatag: Jointly tagging and lemmatizing for morphologically rich languages with brnns. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4921--4928

    2018

  19. [27]

    Daniel Kondratyuk, Tom \'a s Gaven c iak, Milan Straka, and Jan Haji c . 2018 b . Lemmatag: jointly tagging and lemmatizing for morphologically-rich languages with brnns. arXiv preprint arXiv:1808.03703

    2018 arXiv

  20. [28]

    Jarkko Lagus and Arto Klami. 2021. Learning to lemmatize in the word representation space. In Proceedings of the 23rd Nordic Conference on Computational Linguistics (NoDaLiDa), pages 249--258

    2021

  21. [29]

    Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. 2023. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.03281

    2023 arXiv

  22. [30]

    Juan Liberato, Bashar Alhafni, Muhamed Khalil, and Nizar Habash. 2024. https://doi.org/10.18653/v1/2024.arabicnlp-1.5 Strategies for A rabic readability modeling . In Proceedings of The Second Arabic Natural Language Processing Conference, pages 55--66, Bangkok, Thailand. Asso...

    2024 doi

  23. [31]

    Mohamed Maamouri, Ann Bies, Tim Buckwalter, and Wigdan Mekki. 2004. The penn arabic treebank: Building a large-scale annotated arabic corpus. In NEMLAR conference on Arabic language resources and tools, volume 27, pages 466--467. Cairo

    2004

  24. [32]

    Mohamed Maamouri, David Graff, Basma Bouziri, Sondos Krouna, Ann Bies, and Seth Kulick. 2010. Ldc standard arabic morphological analyzer (sama) version 3.1. (No Title)

    2010

  25. [33]

    Quinn McNemar. 1947. https://doi.org/10.1007/bf02295996 Note on the sampling error of the difference between correlated proportions or percentages . Psychometrika, 12(2):153--157

    1947 doi

  26. [34]

    Hamdy Mubarak. 2017. Build fast and accurate lemmatization for arabic. arXiv preprint arXiv:1710.06700

    2017 arXiv

  27. [35]

    Hamdy Mubarak. 2018. https://aclanthology.org/L18-1181 Build fast and accurate lemmatization for A rabic . In Proceedings of the Eleventh International Conference on Language Resources and Evaluation ( LREC 2018) , Miyazaki, Japan. European Language Resources Association (ELRA)

    2018

  28. [36]

    Thomas Muller, Ryan Cotterell, Alexander Fraser, and Hinrich Sch \"u tze. 2024. Joint lemmatization and morphological tagging with lemming. arXiv preprint arXiv:2405.18308

    2024 arXiv

  29. [37]

    Ossama Obeid, Go Inoue, and Nizar Habash. 2022. Camelira: An arabic multi-dialect morphological disambiguator. arXiv preprint arXiv:2211.16807

    2022 arXiv

  30. [38]

    Ossama Obeid, Nasser Zalmout, Salam Khalifa, Dima Taji, Mai Oudah, Bashar Alhafni, Go Inoue, Fadhl Eryani, Alexander Erdmann, and Nizar Habash. 2020. https://aclanthology.org/2020.lrec-1.868 CAM e L tools: An open source python toolkit for A rabic natural language processing ....

    2020

  31. [39]

    Arfath Pasha, Mohamed Al-Badrashiny, Ahmed El Kholy, Ramy Eskander, Mona Diab, Nizar Habash, Manoj Pooleery, Owen Rambow, and Ryan Roth. 2014. Madamira: A fast, comprehensive tool for morphological analysis and disambiguation of arabic. In In Proceedings of LREC

    2014

  32. [40]

    Ryan Roth, Owen Rambow, Nizar Habash, Mona Diab, and Cynthia Rudin. 2008. A rabic morphological tagging, diacritization, and lemmatization using lexeme models and feature ranking. In Proceedings of the Conference of the Association for Computational Linguistics (ACL), Columbus, Ohio

    2008

  33. [41]

    Masoud Jalili Sabet, Philipp Dufter, Fran c ois Yvon, and Hinrich Sch \"u tze. 2020. Simalign: High quality word alignments without parallel training data using static and contextualized embeddings. arXiv preprint arXiv:2004.08728

    2020 arXiv

  34. [42]

    Aleksi Sahala. 2024. Neural lemmatization and pos-tagging models for coptic, demotic and earlier egyptian. In Proceedings of the 1st Workshop on Machine Learning for Ancient Languages (ML4AL 2024), pages 87--97

    2024

  35. [43]

    Djam \'e Seddah, Grzegorz Chrupa a, \"O zlem C etino g lu, Josef van Genabith, and Marie Candito. 2010. https://aclanthology.org/W10-1410 Lemmatization and lexicalized statistical parsing of morphologically-rich languages: the case of F rench . In Proceedings of the NAACL HLT ...

    2010

  36. [44]

    Nasredine Semmar, Meriama Laib, and Christian Fluhr. 2006. A deep linguistic analysis for cross-language information retrieval. In LREC, pages 2507--2510

    2006

  37. [45]

    Rico Sennrich and Barry Haddow. 2016. Linguistic input features improve neural machine translation. arXiv preprint arXiv:1606.02892

    2016 arXiv

  38. [46]

    Dima Taji, Salam Khalifa, Ossama Obeid, Fadhl Eryani, and Nizar Habash. 2018 a . An arabic morphological analyzer and generator with copious features. In Proceedings of the fifteenth workshop on computational research in phonetics, phonology, and morphology, pages 140--150

    2018

  39. [47]

    Dima Taji, Salam Khalifa, Ossama Obeid, Fadhl Eryani, and Nizar Habash. 2018 b . https://doi.org/10.18653/v1/W18-5816 An A rabic morphological analyzer and generator with copious features . In Proceedings of the Fifteenth Workshop on Computational Research in Phonetics, Phonol...

    2018 doi

  40. [48]

    Mustafa Yaseen, Mohammed Attia, Bente Maegaard, Khalid Choukri, Niklas Paulsson, Salah Haamid, Steven Krauwer, Chomicha Bendahman, Hanne Fers e, Mohsen A Rashwan, et al. 2006. Building annotated written and spoken arabic lrs in nemlar project. In LREC, pages 533--538. Citeseer

    2006

  41. [49]

    Nasser Zalmout. 2020. Morphological Tagging and Disambiguation in Dialectal Arabic Using Deep Learning Architectures. Ph.D. thesis, New York University Tandon School of Engineering

    2020

  42. [50]

    Nasser Zalmout and Nizar Habash. 2020. https://doi.org/10.18653/v1/2020.acl-main.736 Joint diacritization, lemmatization, normalization, and fine-grained morphological tagging . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8...

    2020 doi

Pith tools