REVIEW 4 major objections 5 minor 26 references
EDU-NER-2025: Named Entity Recognition in Urdu Educational Texts using XLM-RoBERTa with X (formerly Twitter)
T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The paper claims that EDU-NER-2025 is the first Urdu education-domain NER dataset and that fine-tuned XLM-RoBERTa reaches 98% accuracy on it.
desk verdict A genuine gap in Urdu educational NER, but the paper's own arithmetic is contradictory and the dataset is unreleased, so the 98% claim cannot be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the dataset itself—EDU-NER-2025, a Twitter-derived Urdu corpus of education-related tweets with 13 entity labels including PERSON, ORGANIZATION, LOCATION, DESIGNATION, BOOK, COURSE, EVENT, DATE, TIME, NUMBER, and OTHER. The annotation process is central: eight native-speaker annotators were reduced in two rounds to three consistent ones, and Fleiss' kappa of 0.79 is used as evidence of substantial agreement. The model-side machinery is transfer learning from XLM-RoBERTa-base, a multilingual transformer pretrained on 100 languages, fine-tuned with 128-token sequences, batch size 16, learning rate 2e-5, AdamW, 500 warmup steps, and 5 epochs under 5-fold cross-validation.
What would settle it
A reader could settle the central claim by taking the released EDU-NER-2025 and checking three arithmetic identities: the tweet count (reported 18,455), total words (660,140), and the sum of per-label support (829,589) must be reconcilable under the paper's preprocessing; if they are not, the corpus statistics and the 98% benchmark rest on an unstable foundation. Re-running the paper's XLM-RoBERTa configuration on the released corpus and comparing per-class F1 on the 5,765-support EVENT class would show whether the headline accuracy actually holds for the rarest entity type.
Extended reading notes
Core claim
The paper's central claim is that EDU-NER-2025 is the first Urdu NER dataset built specifically for the education domain, and that a fine-tuned XLM-RoBERTa-base model labels its entities almost perfectly: 0.98 precision, recall, F1, and cross-validation score across 5 folds. The claim extends to the broader point that contextual multilingual embeddings overcome domain-adaptation problems for Urdu NER without hand-built gazetteers, since the same task that classical machine learning solves at 0.89 and BiLSTM-with-FastText at 0.85 is solved at 0.98 by transfer learning. The paper reports class-wise F1 between 0.92 (LOCATION) and 0.99 (COURSE, EVENT, OTHER), with the OTHER class dominating the support at 635,387 instances.
Load-bearing premise
The result stands on the assumption that EDU-NER-2025 is a trustworthy gold-standard corpus, but the paper's reported tweet, word, and label counts do not match each other and the dataset has not been released for verification.
Editorial extensions
If this is right
- If EDU-NER-2025 is accepted as a gold standard, then domain-specific Urdu NER is essentially solved by generic multilingual transfer learning, at least for Twitter-style educational text.
- The 13-entity schema gives downstream Urdu educational NLP a ready-made tagging standard for academic roles, institutions, courses, and events.
- The paper's annotation pipeline—two-round annotator screening, Fleiss' kappa checks, and guideline iteration—can be lifted directly for low-resource domains beyond education.
- The reported 98% figure implies that classical machine-learning baselines (RF at 89%) and static-embedding deep models (85%) are no longer competitive for this task, so future work should start from transformers.
Reading between the lines
- If the 98% result replicates on a cleaned release, the real bottleneck for low-resource NER is dataset construction, not model architecture: the labeling effort, not the learner, is what unlocks the domain.
- Because the OTHER class holds over 76% of the labeled tokens, the headline accuracy mostly reflects non-entity tagging; a fair comparison of entity detection should be weighted by the per-class supports reported in the paper's class-wise table.
- A direct test not run in the paper is to hold out one complete entity type, such as COURSE, during fine-tuning and measure zero-shot recovery; that would show whether the model learns the category or memorizes the lexicon.
- The paper's plan to extend the corpus to academic reports, lecture transcripts, and textbooks implies a testable prediction: performance will drop on formal genres if the model overfits Twitter style, and annotation guidelines will need genre-specific additions.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces EDU-NER-2025, described as the first manually annotated Urdu Named Entity Recognition (NER) dataset in the education domain, sourced from Twitter. The authors report an annotation procedure with three retained annotators, a Fleiss' Kappa of 79%, and a label set that is described inconsistently as 11 or 13 entity types. They evaluate three classical machine learning models (SVM, LR, RF), two deep learning architectures (CNN, BiLSTM with FastText and GloVe embeddings), and three transformer-based models (BERT, RoBERTa, XLM-RoBERTa) under 5-fold cross-validation, claiming that fine-tuned XLM-RoBERTa achieves 98% precision, recall, F1, and cross-validation accuracy, surpassing the best classical approach (RF) by 10.11 percentage points. The central claims are the novelty of the dataset, the completeness of the annotation guidelines, and the strong empirical performance of XLM-RoBERTa.
Significance. If the dataset were real, reliable, and publicly available, this work would address a genuine resource gap for Urdu NER in a domain-specific setting, and the systematic comparison of classical, deep, and transformer models would be a useful empirical reference. The authors deserve credit for providing detailed annotation guidelines, measuring inter-annotator agreement, and evaluating a broad set of models. However, the claimed contribution cannot currently be assessed because the dataset is not released and the reported corpus statistics are internally contradictory: the sum of per-label supports in Table 6 does not match the reported total word or token counts, the tweet count differs by an order of magnitude between the methodology and Figure 5, and the number of entity types changes between 13 and 11. These inconsistencies strike at the only object the 98% result is about, making the central empirical claim unverifiable as presented.
major comments (4)
- [Corpus characteristics and standardization; Table 6] The class-wise support values in Table 6 sum to 829,589 labels (BOOK 13,389 + COURSE 16,792 + DATE 6,162 + DESIGNATION 31,351 + EVENT 5,765 + LOCATION 19,823 + NUMBER 7,502 + ORGANIZATION 46,689 + OTHER 635,387 + PERSON 40,646 + TIME 6,083). In a standard token-level NER setup, every token receives exactly one label, so this sum must equal the total number of labeled tokens. The paper simultaneously reports 660,140 total words in Figure 5 and 'Total Tokens' of nearly 599,000 in the text accompanying Figure 4. The discrepancy of roughly 169,000 to 230,000 labels is impossible unless non-exclusive or nested annotations were used, which is not stated. This inconsistency makes the reported 98% accuracy unverifiable and is a load-bearing flaw in the paper's central claim.
- [Construction of dataset; Figure 5] The methodology states that the authors 'utilized the Twitter API to retrieve nearly 30,0000 Urdu tweets related to education,' which appears to claim approximately 300,000 tweets (and is at minimum ambiguous due to the extra zero). Figure 5, however, reports a dataset of 18,455 tweets, and the text describing Figure 5 repeats this number. A factor-of-ten discrepancy in the corpus size is not a cosmetic typo: it directly affects every derived statistic, including average words per tweet and the token totals, and prevents a reader from knowing the actual scale of the dataset.
- [Abstract, Literature Review, Methodology, Table 6] The paper is internally inconsistent about the number of entity types. The abstract and Literature Review claim 13 distinct named entity classes, while the Methodology states that 'each tweet was then labeled into one of 11 categories,' and Table 6 lists exactly 11 labels (BOOK, COURSE, DATE, DESIGNATION, EVENT, LOCATION, NUMBER, ORGANIZATION, OTHER, PERSON, TIME). No explanation or reconciliation of this 13-versus-11 discrepancy is provided anywhere in the manuscript.
- [Annotation procedure and Inter Annotator Agreement] The annotator selection process is circular: three annotators were chosen because they 'consistently provided accurate and correct labels,' but the criterion for accuracy appears to be agreement with the labels produced by the other annotators, with no external gold standard. The subsequent Fleiss' Kappa of 79% is then computed on the annotations produced by the same selected annotators, which can inflate agreement. The paper should clarify how a ground truth was established for the selection rounds and whether the reported kappa is computed on a held-out set or on the same data used for selection.
minor comments (5)
- [Throughout] The dataset is referred to inconsistently as 'EDU-NER-2025' and 'Edu-UNER-2025'; the name should be unified.
- [Abstract and Introduction] There are typos and unclear phrases, including 'XLR-RoBERTa' instead of 'XLM-RoBERTa', 'foure' instead of 'four', and an incomplete citation in the Introduction ('question answering []').
- [Figure and table numbering] Several figures and tables share numbers or are mislabeled: there are multiple Figures 3 and 4, the 'Top performing models' figure is labeled Figure 2 while the data preprocessing figure also appears as Figure 2, and Table 4 appears twice for different content.
- [Transfer learning Results] The results for BERT, RoBERTa, and XLM-RoBERTa are reported as uniform scores of 0.97, 0.96, and 0.98 across precision, recall, F1, and C.V. score, with no variance or per-fold results; reporting only the mean of a 5-fold cross-validation without standard deviation makes the comparisons difficult to interpret.
- [References] Reference [1] is cited in the Introduction as the source of a generic news-based Urdu NER corpus, but the reference list entry is a survey on NER in historical documents, which does not appear to match the citation context.
Circularity Check
No significant circularity: the paper's empirical pipeline is self-contained, though internal dataset statistics are inconsistent and unverifiable.
full rationale
The paper's central claim is empirical: it constructs a manually annotated Urdu educational NER dataset (EDU-NER-2025), applies standard preprocessing, fine-tunes XLM-RoBERTa along with other models, and reports 5-fold cross-validation scores. There is no derivation chain in which a predicted quantity is defined in terms of the fitted quantity, and no equation reduces the reported 98% accuracy to the annotation statistics by construction. XLM-RoBERTa is an existing public multilingual transformer, not a model whose parameters are derived from the claimed outputs. The self-citations in the reference list (e.g., [2], [9], [22], [24]) appear as related work or prior applications of similar methods and are not load-bearing justifications for the novelty claim or for the performance result. No uniqueness theorem from prior work by the same authors is invoked to declare the model choice forced. The paper does, however, contain serious internal numerical inconsistencies: the class-wise support values in Table 6 sum to 829,589 while the text reports total tokens near 599,000 and total words of 660,140; it also reports both 18,455 tweets and 'nearly 30,0000' tweets, and both 13 and 11 entity types. These inconsistencies make the reported 98% result unverifiable, but they are evidence of data-quality or reporting problems, not of circular reasoning. The central derivation is therefore self-contained, so the circularity score is low.
Assumptions & free parameters
free parameters (2)
- Transformer fine-tuning hyperparameters =
learning rate 2e-5, batch size 16, 5 epochs, warmup 500, weight decay 0.01
- ML/DL hyperparameters =
RF n_estimators=100; SVM C=1.0, gamma='scale'; LR C=1.0; BiLSTM/CNN hidden units=128, epochs=10, batch=32, lr=0.001
assumptions (3)
- domain assumption The EDU-NER-2025 annotations are a valid gold standard for Urdu educational NER.
- domain assumption The Twitter search and preprocessing pipeline yields a representative corpus of educational Urdu text.
- domain assumption Token-level 5-fold cross-validation measures real-world NER generalization.
Cite this review
Pith. "Pith review of EDU-NER-2025: Named Entity Recognition in Urdu Educational Texts using XLM-RoBERTa with X (formerly Twitter)." pith.science (2026). https://pith.science/paper/LOBVJGCN
@misc{pith2026250418142,
author = {Pith},
title = {Pith review of: EDU-NER-2025: Named Entity Recognition in Urdu Educational Texts using XLM-RoBERTa with X (formerly Twitter)},
year = {2026},
howpublished = {\url{https://pith.science/paper/LOBVJGCN}},
note = {Machine review of arXiv:2504.18142}
}
read the original abstract
Named Entity Recognition (NER) plays a pivotal role in various Natural Language Processing (NLP) tasks by identifying and classifying named entities (NEs) from unstructured data into predefined categories such as person, organization, location, date, and time. While extensive research exists for high-resource languages and general domains, NER in Urdu particularly within domain-specific contexts like education remains significantly underexplored. This is Due to lack of annotated datasets for educational content which limits the ability of existing models to accurately identify entities such as academic roles, course names, and institutional terms, underscoring the urgent need for targeted resources in this domain. To the best of our knowledge, no dataset exists in the domain of the Urdu language for this purpose. To achieve this objective this study makes three key contributions. Firstly, we created a manually annotated dataset in the education domain, named EDU-NER-2025, which contains 13 unique most important entities related to education domain. Second, we describe our annotation process and guidelines in detail and discuss the challenges of labelling EDU-NER-2025 dataset. Third, we addressed and analyzed key linguistic challenges, such as morphological complexity and ambiguity, which are prevalent in formal Urdu texts.
Figures
Figures from the paper (7 more)
Reference graph
Works this paper leans on
-
[1]
NER and classification in historical documents: A survey
Ehrmann, Maud, Ahmed Hamdi, Elvys Linhares Pontes, Matteo Romanello, and Antoine Doucet. "NER and classification in historical documents: A survey." ACM Computing Surveys 56, no. 2 (2023): 1-47
work page 2023
-
[2]
Ullah, Fida, Alexander Gelbukh, Muhammad Tayyab Zamir, Edgardo Manuel Felipe Riverόn, and Grigori Sidorov. "Enhancement of NER in Low-Resource Languages with Data Augmentation and BERT Models: A Case Study on Urdu." Computers 13, no. 10 (2024): 258
work page 2024
-
[3]
Core techniques of question answering syst ems over knowl edgebases: a survey
Diefenbach D, Lopez V, Singh K, Maret P. Core techniques of question answering syst ems over knowl edgebases: a survey. Knowledge and information systems. 2018; 55(3):529 –569. https://doi.org/10. 1007/s10115-017-1100-y
work page 2018
-
[4]
Qadatasetexplosion: A taxonomy of nlp resources for question answering and reading compr ehension
RogersA, GardnerM, Augenstein I. Qadatasetexplosion: A taxonomy of nlp resources for question answering and reading compr ehension. ACMComputingSurveys.2023;55(10):1 –45. https://doi.org/ 10.1145/3560260
doi:10.1145/3560260 2023
-
[5]
arXiv preprint arXiv:191007475
Lewis P, OğuzB, RinottR, RiedelS, SchwenkH.MLQA: Evaluatingcross -lingual extractive question answering. arXiv preprint arXiv:191007475. 2019
work page 2019
-
[6]
Haq, R., Zhang, X., Khan, W., & Feng, Z. (2023). Urdu named entity recognition system using deep learning approaches. The Computer Journal, 66(8), 1856-1869
work page 2023
-
[7]
Deep recurrent neural networks with word embeddings for Urdu NER
Khan, Wahab, Ali Daud, Fahd Alotaibi, Naif Aljohani, and Sachi Arafat. "Deep recurrent neural networks with word embeddings for Urdu NER." ETRI Journal 42, no. 1 (2020): 90-100
work page 2020
-
[8]
Kanwal, S., Malik, K., Shahzad, K., Aslam, F., & Nawaz, Z. (2019). Urdu named entity recognition: Corpus generation and deep learning applications. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 19(1), 1-13
work page 2019
Show all 26 references
-
[9]
Urdu NER with attention bi -lstm-crf model
Ullah, Fida, Ihsan Ullah, and Olga Kolesnikova. "Urdu NER with attention bi -lstm-crf model." In Mexican International Conference on Artificial Intelligence , pp. 3-17. Cham: Springer Nature Switzerland, 2022
2022
-
[10]
Santana, B., Campos, R., Amorim, E., Jorge, A., Silvano, P., & Nunes, S. (2023). A survey on narrative extraction from textual data. Artificial Intelligence Review, 1-43
2023
-
[11]
Mulcaire, P., Kasai, J., & Smith, N. A. (2019). Polyglot contextual representations improv e crosslingual transfer. arXiv preprint arXiv:1902.09697
2019 arXiv
-
[12]
N., Rabbi, J., Masud, M., Hasan, M
Sen, O., Fuad, M., Islam, M. N., Rabbi, J., Masud, M., Hasan, M. K., ... & Iftee, M. A. R. (2022). Bangla natural language processing: A comprehensive analysis of classical, machine learning, and deep learning-based methods. IEEE Access, 10, 38999-39044
2022
-
[13]
Malik, M. K. (2017). Urdu NER and classification system using artificial neural network. ACM Transactions on Asian and Low-Resource Language Information Processing (TALLIP), 17(1), 1- 13
2017
-
[14]
Mukund, S., & Srihari, R. K. 2009. NE tagging for Urdu based on bootstrap POS learning. In Proceedings of the Third International Workshop on Cross Lingual Information Access: Addressing the Information Need of Multilingual Societies (CLIAWS3) (pp. 61-69)
2009
-
[15]
Mukund, S., Srihari, R., & Peterson, E. (2010). An information -extraction system for Urdu ---a resource-poor language. ACM Transactions on Asian Language Information Processing (TALIP), 9(4), 1-43
2010
-
[16]
B., Zaharakis, I
Kotsiantis, S. B., Zaharakis, I. D., & Pintelas, P. E. (2006). Machine learning: a review of classification and combining techniques. Artificial Intelligence Review, 26, 159-190
2006
-
[17]
Khan, W., Daud, A., Shahzad, K., Amjad, T., Banjar, A., & Fasihuddin, H. (2022). NER Using Conditional Random Fields. Applied Sciences, 12(13), 6391
2022
-
[18]
P.: A hybrid Oriya NER system: Integrating HMM with MaxEnt
Biswas, S., Mohanty, S., Mishra, S. P.: A hybrid Oriya NER system: Integrating HMM with MaxEnt. In: 2009 Second International Conference on Emerging Trends in Engineering & Technology, pp. 639–643. IEEE (2009)
2009
-
[19]
Gupta, V., & Joshi, N. (2022). Identification and extraction of multiword expressions from Hindi & Urdu language in natural language processing. International Journal of Advanced Technology and Engineering Exploration, 9(91), 807
2022
-
[20]
Fraser, P. D. J. (2009). English: The prototypical world language for the twenty first century. Lulu. com
2009
-
[21]
Singh, U., Goyal, V., & Lehal, G. S. (2012, December). Named entity recognition system for Urdu. In Proceedings of COLING 2012 (pp. 2507-2518)
2012
-
[22]
T., Ahmad, M., Sidorov, G., & Gelbukh, A
Ullah, F., Zamir, M. T., Ahmad, M., Sidorov, G., & Gelbukh, A. (2024). Hope: A multilingual approach to identifying positive communication in social media. In Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2024), co-located with the 40th Conference of the Spani...
2024
-
[23]
& Batyrshin, I
Ahmad, M., Usman, S., Farid, H., Ameer, I., Muzammil, M., Hamza, A., ... & Batyrshin, I. (2024). Hope Speech Detection Using Social Media Discourse (Posi -Vox-2024): A Trans fer Learning Approach. Journal of Language and Education, 10(4 (40)), 31-43
2024
-
[24]
Usman, M., Ahmad, M., Ullah, F., Muzamil, M., Hamza, A., Jalal, M., & Gelbukh, A. (2025). Fine - Tuned RoBERTa Model for Bug Detection in Mobile Games: A Comprehensive Approach. Computers, 14(4), 113
2025
-
[25]
Ahmad, M., Farid, H., Ameer, I., Muzamil, M., Jalal, A. H. M., Batyrshin, I., & Sidorov, G. (2025). Opioid Named Entity Recognition (ONER-2025) from Reddit. arXiv preprint arXiv:2504.00027
2025 arXiv
-
[26]
& Sidorov, G
Ahmad, M., Ameer, I., Sharif, W., Usman, S., Muzamil, M., Hamza, A., ... & Sidorov, G. (2025). Multilingual hope speech detection from tweets using transfer learning models. Scientific Reports, 15(1), 9005
2025
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.