{"id":"63e8f8f6-88e8-444e-a479-d468dd72210e","arxiv_id":"2504.18142","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A claimed new Urdu educational NER dataset and benchmark reports 98% accuracy for XLM-RoBERTa, but the dataset is unavailable and several reported statistics are inconsistent.","lead":"The paper presents EDU-NER-2025, a manually annotated Urdu dataset for educational named entities, and reports that fine-tuned XLM-RoBERTa reaches 98% cross-validation accuracy. A generalist might read this as an example of low-resource NLP resource building, but the dataset is not released and the paper's own statistics contradict each other.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 6 support sums to 829,589 labels while total tokens are reported as 599,000 or 660,140; since each token receives exactly one NER label, the reported numbers cannot all be correct, so the 98% claim is unverifiable.","rationale":"The reader's verdict is REJECT, and my stress-test supports that verdict without modification. The load-bearing premise is that EDU-NER-2025 is a reliable, internally consistent gold-standard dataset. The paper's own Table 6, Figure 4, and Figure 5 contradict this premise: the sum of class-wise supports exceeds the reported total token counts by a wide margin, which is impossible for standard token-level NER where each token is assigned exactly one label. This is a concrete, numeric inconsistency in the central artifact, not a stylistic issue. The claimed 98% F1 cannot be evaluated because the evaluation's substrate is not coherent as reported. I also note the unreleased dataset, the mismatch between 'nearly 30,0000' tweets and 18,455 tweets, and the 13-vs-11 entity-type counts. Any one of these would warrant caution; together they make the headline claim unsupported. I am not accusing the authors of dishonesty; I am stating that the manuscript's internal arithmetic is inconsistent and that no external verification is currently possible. The proposed concrete test—recompute support and token counts from a released corpus—would settle whether the concern lands. If the released data matches the reported totals exactly, the inconsistency would be resolved by correcting the paper's prose, but as written the numbers cannot all be true. A resubmission with the dataset, corrected statistics, and a majority-class baseline comparison could meet the standard for a conditional accept; the current version does not.","tokens_in":11191,"tokens_out":3412,"duration_ms":37083,"concrete_test":"Obtain the released EDU-NER-2025 dataset (or, failing release, ask the authors for the raw corpus and the exact tokenization/annotation script) and recompute the per-label support and total token count. If the sum of per-label supports differs from the reported total tokens by more than one token per ten thousand, then the evaluation setup as described cannot be correct; the 98% result would need to be re-derived on a consistent, released corpus before the central claim can be accepted.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim is that fine-tuned XLM-RoBERTa achieves 98% accuracy/F1 on EDU-NER-2025 under 5-fold cross-validation. This claim is only meaningful if EDU-NER-2025 is a real, internally consistent token-level NER corpus. That precondition fails. In Table 6, the class-wise support values sum to 829,589 (Book 13,389 + Course 16,792 + Date 6,162 + Designation 31,351 + Event 5,765 + Location 19,823 + Number 7,502 + Organization 46,689 + Other 635,387 + Person 40,646 + Time 6,083). The paper simultaneously reports 660,140 total words (Figure 5) and 'Total Tokens' of nearly 599,000 (Figure 4). In a standard BIO/IOB token-level NER setup, every token receives exactly one label, so the sum of supports must equal the total number of tokens. The discrepancy of roughly 169,000–230,000 labels is impossible unless the labels are not mutually exclusive, nested annotations are used, or the reported totals are fabricated or miscomputed. No such caveat is stated. Moreover, the methodology says 'nearly 30,0000' Urdu tweets were retrieved, while Figure 5 reports 18,455 tweets; the number of entity types is given as both 13 and 11; and the dataset is not released. These are not cosmetic typos: they strike at the only object the 98% result is about. A reader cannot distinguish 'high accuracy on a well-formed dataset' from 'high accuracy on an internally inconsistent or misreported dataset,' and the dataset's absence forecloses any external check. The novelty and resource claims therefore rest on the same unsupported foundation. I do not claim fraud; I claim the paper, as written, does not establish its central result because its own numbers contradict token-level NER semantics.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces EDU-NER-2025, described as the first manually annotated Urdu Named Entity Recognition (NER) dataset in the education domain, sourced from Twitter. The authors report an annotation procedure with three retained annotators, a Fleiss' Kappa of 79%, and a label set that is described inconsistently as 11 or 13 entity types. They evaluate three classical machine learning models (SVM, LR, RF), two deep learning architectures (CNN, BiLSTM with FastText and GloVe embeddings), and three transformer-based models (BERT, RoBERTa, XLM-RoBERTa) under 5-fold cross-validation, claiming that fine-tuned XLM-RoBERTa achieves 98% precision, recall, F1, and cross-validation accuracy, surpassing the best classical approach (RF) by 10.11 percentage points. The central claims are the novelty of the dataset, the completeness of the annotation guidelines, and the strong empirical performance of XLM-RoBERTa.","tokens_in":11569,"tokens_out":3273,"duration_ms":32449,"significance":"If the dataset were real, reliable, and publicly available, this work would address a genuine resource gap for Urdu NER in a domain-specific setting, and the systematic comparison of classical, deep, and transformer models would be a useful empirical reference. The authors deserve credit for providing detailed annotation guidelines, measuring inter-annotator agreement, and evaluating a broad set of models. However, the claimed contribution cannot currently be assessed because the dataset is not released and the reported corpus statistics are internally contradictory: the sum of per-label supports in Table 6 does not match the reported total word or token counts, the tweet count differs by an order of magnitude between the methodology and Figure 5, and the number of entity types changes between 13 and 11. These inconsistencies strike at the only object the 98% result is about, making the central empirical claim unverifiable as presented.","major_comments":[{"comment":"The class-wise support values in Table 6 sum to 829,589 labels (BOOK 13,389 + COURSE 16,792 + DATE 6,162 + DESIGNATION 31,351 + EVENT 5,765 + LOCATION 19,823 + NUMBER 7,502 + ORGANIZATION 46,689 + OTHER 635,387 + PERSON 40,646 + TIME 6,083). In a standard token-level NER setup, every token receives exactly one label, so this sum must equal the total number of labeled tokens. The paper simultaneously reports 660,140 total words in Figure 5 and 'Total Tokens' of nearly 599,000 in the text accompanying Figure 4. The discrepancy of roughly 169,000 to 230,000 labels is impossible unless non-exclusive or nested annotations were used, which is not stated. This inconsistency makes the reported 98% accuracy unverifiable and is a load-bearing flaw in the paper's central claim.","section":"Corpus characteristics and standardization; Table 6"},{"comment":"The methodology states that the authors 'utilized the Twitter API to retrieve nearly 30,0000 Urdu tweets related to education,' which appears to claim approximately 300,000 tweets (and is at minimum ambiguous due to the extra zero). Figure 5, however, reports a dataset of 18,455 tweets, and the text describing Figure 5 repeats this number. A factor-of-ten discrepancy in the corpus size is not a cosmetic typo: it directly affects every derived statistic, including average words per tweet and the token totals, and prevents a reader from knowing the actual scale of the dataset.","section":"Construction of dataset; Figure 5"},{"comment":"The paper is internally inconsistent about the number of entity types. The abstract and Literature Review claim 13 distinct named entity classes, while the Methodology states that 'each tweet was then labeled into one of 11 categories,' and Table 6 lists exactly 11 labels (BOOK, COURSE, DATE, DESIGNATION, EVENT, LOCATION, NUMBER, ORGANIZATION, OTHER, PERSON, TIME). No explanation or reconciliation of this 13-versus-11 discrepancy is provided anywhere in the manuscript.","section":"Abstract, Literature Review, Methodology, Table 6"},{"comment":"The annotator selection process is circular: three annotators were chosen because they 'consistently provided accurate and correct labels,' but the criterion for accuracy appears to be agreement with the labels produced by the other annotators, with no external gold standard. The subsequent Fleiss' Kappa of 79% is then computed on the annotations produced by the same selected annotators, which can inflate agreement. The paper should clarify how a ground truth was established for the selection rounds and whether the reported kappa is computed on a held-out set or on the same data used for selection.","section":"Annotation procedure and Inter Annotator Agreement"}],"minor_comments":[{"comment":"The dataset is referred to inconsistently as 'EDU-NER-2025' and 'Edu-UNER-2025'; the name should be unified.","section":"Throughout"},{"comment":"There are typos and unclear phrases, including 'XLR-RoBERTa' instead of 'XLM-RoBERTa', 'foure' instead of 'four', and an incomplete citation in the Introduction ('question answering []').","section":"Abstract and Introduction"},{"comment":"Several figures and tables share numbers or are mislabeled: there are multiple Figures 3 and 4, the 'Top performing models' figure is labeled Figure 2 while the data preprocessing figure also appears as Figure 2, and Table 4 appears twice for different content.","section":"Figure and table numbering"},{"comment":"The results for BERT, RoBERTa, and XLM-RoBERTa are reported as uniform scores of 0.97, 0.96, and 0.98 across precision, recall, F1, and C.V. score, with no variance or per-fold results; reporting only the mean of a 5-fold cross-validation without standard deviation makes the comparisons difficult to interpret.","section":"Transfer learning Results"},{"comment":"Reference [1] is cited in the Introduction as the source of a generic news-based Urdu NER corpus, but the reference list entry is a survey on NER in historical documents, which does not appear to match the citation context.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper's central contribution is a dataset plus an empirical evaluation, but the dataset is not released and the reported corpus statistics are mutually contradictory (Table 6 support sum 829,589 vs Figure 5 total words 660,140 vs Figure 4 'Total Tokens' ~599,000; tweet count 'nearly 30,0000' vs 18,455; 13 vs 11 entity types). These are not presentation issues; they make the 98% accuracy claim unverifiable. If the authors can release the dataset and correct the statistics, a substantially revised version might merit reconsideration, but as it stands the manuscript does not meet the bar for publication. I would also ask the editor to verify the novelty claim about the 'first' Urdu educational NER dataset, since the literature review is selective and does not discuss recent Urdu NER resources that may overlap."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis one is a useful reminder that a good idea and careful annotation description don't survive arithmetic contradictions. The paper fills a real gap—Urdu NER has no education-domain dataset—and the authors describe their annotation process with more transparency than usual: eight annotators winnowed to three, Fleiss' kappa 0.79, guidelines with examples. The experimental sweep is broad (traditional ML, BiLSTM/CNN, three transformers). That is genuine effort.\n\nThe problem is that the central numbers can't all be true. Table 6's support column sums to 829,589 labels; the paper reports total tokens of ~599,000 and total words of 660,140. In token-level NER each token gets exactly one label, so the sum of supports must equal token count. It doesn't. The abstract says 13 entity types; Table 6 lists 11. Corpus size is 'nearly 30,0000' tweets (presumably 300,000) while Figure 5 says 18,455. The dataset is not released. So the headline 98% for XLM-RoBERTa is unverifiable—it could be a great result on a good dataset, or a number generated from a misreported one. Nothing in the manuscript lets you distinguish those.\n\nThe modeling side is routine fine-tuning, with an odd choice of English-only BERT/RoBERTa baselines on Urdu. There's no downstream application, so significance is modest. The strongest part is the linguistic discussion of morphological ambiguity and code-mixing, which suggests the authors know the domain.\n\nI would not put this in front of reviewers yet. A serious editor should send it back: ask for the dataset to be posted, the statistics reconciled, and the class count fixed. If the data is real and the numbers hold up after correction, a resubmission would be a welcome resource for the low-resource community. As written, it doesn't pass a basic sanity check.\n\nRecommendation: desk reject in current form; invite resubmission after data release and correction.","headline":"A genuine gap in Urdu educational NER, but the paper's own arithmetic is contradictory and the dataset is unreleased, so the 98% claim cannot be trusted.","tokens_in":12150,"tokens_out":3808,"would_cite":false,"duration_ms":38999,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that EDU-NER-2025 is the first Urdu education-domain NER dataset and that fine-tuned XLM-RoBERTa reaches 98% accuracy on it.","keywords":["Urdu named entity recognition","education domain dataset","XLM-RoBERTa","Twitter corpus","low-resource NLP","transfer learning","annotation guidelines","5-fold cross-validation"],"falsifier":"A reader could settle the central claim by taking the released EDU-NER-2025 and checking three arithmetic identities: the tweet count (reported 18,455), total words (660,140), and the sum of per-label support (829,589) must be reconcilable under the paper's preprocessing; if they are not, the corpus statistics and the 98% benchmark rest on an unstable foundation. Re-running the paper's XLM-RoBERTa configuration on the released corpus and comparing per-class F1 on the 5,765-support EVENT class would show whether the headline accuracy actually holds for the rarest entity type.","tokens_in":11003,"feed_emoji":"🎓","tokens_out":6432,"duration_ms":59585,"temperature":0.7,"pith_summary":"This paper sets out to fill a gap it says no one else has addressed: named entity recognition in the Urdu education domain. It introduces EDU-NER-2025, a manually annotated corpus of roughly 18,455 education-related Urdu tweets tagged with 13 entity types, and documents the annotation process and its linguistic challenges. On this corpus the paper compares ten model configurations—classical classifiers, BiLSTM and CNN with static embeddings, and three transformer models—and reports that fine-tuned XLM-RoBERTa reaches 98% precision, recall, and F1 under 5-fold cross-validation, beating the best classical model (Random Forest, 89%) by 10.11 points. A sympathetic reader would care because high-accuracy NER for an under-resourced language in a domain-specific setting would open the door to downstream educational NLP tools.","feed_headline":"First Urdu education NER dataset, 98% accuracy via XLM-RoBERTa","feed_subtitle":"A new 18,455-tweet Urdu education corpus with 13 entity types pushes NER accuracy from 89% to 98%.","key_machinery":"The load-bearing object is the dataset itself—EDU-NER-2025, a Twitter-derived Urdu corpus of education-related tweets with 13 entity labels including PERSON, ORGANIZATION, LOCATION, DESIGNATION, BOOK, COURSE, EVENT, DATE, TIME, NUMBER, and OTHER. The annotation process is central: eight native-speaker annotators were reduced in two rounds to three consistent ones, and Fleiss' kappa of 0.79 is used as evidence of substantial agreement. The model-side machinery is transfer learning from XLM-RoBERTa-base, a multilingual transformer pretrained on 100 languages, fine-tuned with 128-token sequences, batch size 16, learning rate 2e-5, AdamW, 500 warmup steps, and 5 epochs under 5-fold cross-validation.","core_discovery":"The paper's central claim is that EDU-NER-2025 is the first Urdu NER dataset built specifically for the education domain, and that a fine-tuned XLM-RoBERTa-base model labels its entities almost perfectly: 0.98 precision, recall, F1, and cross-validation score across 5 folds. The claim extends to the broader point that contextual multilingual embeddings overcome domain-adaptation problems for Urdu NER without hand-built gazetteers, since the same task that classical machine learning solves at 0.89 and BiLSTM-with-FastText at 0.85 is solved at 0.98 by transfer learning. The paper reports class-wise F1 between 0.92 (LOCATION) and 0.99 (COURSE, EVENT, OTHER), with the OTHER class dominating the support at 635,387 instances.","pith_inferences":["If the 98% result replicates on a cleaned release, the real bottleneck for low-resource NER is dataset construction, not model architecture: the labeling effort, not the learner, is what unlocks the domain.","Because the OTHER class holds over 76% of the labeled tokens, the headline accuracy mostly reflects non-entity tagging; a fair comparison of entity detection should be weighted by the per-class supports reported in the paper's class-wise table.","A direct test not run in the paper is to hold out one complete entity type, such as COURSE, during fine-tuning and measure zero-shot recovery; that would show whether the model learns the category or memorizes the lexicon.","The paper's plan to extend the corpus to academic reports, lecture transcripts, and textbooks implies a testable prediction: performance will drop on formal genres if the model overfits Twitter style, and annotation guidelines will need genre-specific additions."],"forward_implications":["If EDU-NER-2025 is accepted as a gold standard, then domain-specific Urdu NER is essentially solved by generic multilingual transfer learning, at least for Twitter-style educational text.","The 13-entity schema gives downstream Urdu educational NLP a ready-made tagging standard for academic roles, institutions, courses, and events.","The paper's annotation pipeline—two-round annotator screening, Fleiss' kappa checks, and guideline iteration—can be lifted directly for low-resource domains beyond education.","The reported 98% figure implies that classical machine-learning baselines (RF at 89%) and static-embedding deep models (85%) are no longer competitive for this task, so future work should start from transformers."],"supporting_citations":[{"why":"Supplies the task definition and the standard entity categories (person, organization, location, date, time) that the paper's schema builds on.","marker":"[1]"},{"why":"Establishes prior Urdu NER work with data augmentation and BERT, the immediate line of research that EDU-NER-2025 extends into the education domain.","marker":"[2]"},{"why":"Provides a deep-learning Urdu NER baseline that the paper positions against and improves upon.","marker":"[6]"},{"why":"Shows deep recurrent networks with word embeddings for Urdu NER, another prior approach the paper contrasts with transformers.","marker":"[7]"},{"why":"Documents the largest existing Urdu NER dataset and 32 embedding-model experiments, the main resource-scarcity benchmark the paper claims to complement.","marker":"[8]"},{"why":"Introduces an earlier Urdu NER corpus and system (KPU-NE with ANN and HMM), giving historical context for the new dataset.","marker":"[13]"},{"why":"Presents a CRF-based Urdu NER method with the UNER-I dataset, representing the classical supervised approach whose 89% level the transformer surpasses.","marker":"[17]"},{"why":"Describes a rule-based Urdu NER system, motivating the shift to machine learning and transfer learning that the paper advocates.","marker":"[21]"}],"fun_headline_variants":["XLM-RoBERTa hits 98% F1 on first Urdu education NER corpus","New Urdu education NER dataset: RoBERTa beats classic ML by 9 points","Urdu education NER: 13 entity types, 98% F1 with XLM-RoBERTa","First Urdu ED-NER dataset: 18k tweets, 98% F1 via transfer learning","From 89% to 98%: XLM-RoBERTa dominates Urdu education NER"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The result stands on the assumption that EDU-NER-2025 is a trustworthy gold-standard corpus, but the paper's reported tweet, word, and label counts do not match each other and the dataset has not been released for verification.","fun_headline_variants_meta":{"raw":{"variants":["XLM-RoBERTa hits 98% F1 on first Urdu education NER corpus","New Urdu education NER dataset: RoBERTa beats classic ML by 9 points","Urdu education NER: 13 entity types, 98% F1 with XLM-RoBERTa","First Urdu ED-NER dataset: 18k tweets, 98% F1 via transfer learning","From 89% to 98%: XLM-RoBERTa dominates Urdu education NER"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000825,"raw_usage":{"total_tokens":3612,"prompt_tokens":957,"completion_tokens":2655,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":573,"completion_tokens_details":{"reasoning_tokens":2530}},"tokens_in":573,"tokens_out":2655,"duration_ms":17084,"temperature":1.0,"reasoning_tokens":2530,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:22:51.004539+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could settle the central claim by taking the released EDU-NER-2025 and checking three arithmetic identities: the tweet count (reported 18,455), total words (660,140), and the sum of per-label support (829,589) must be reconcilable under the paper's preprocessing; if they are not, the corpus statistics and the 98% benchmark rest on an unstable foundation. Re-running the paper's XLM-RoBERTa configuration on the released corpus and comparing per-class F1 on the 5,765-support EVENT class would show whether the headline accuracy actually holds for the rarest entity type.","supporting_citations":[{"cited_title":"NER and classification in historical documents: A survey","cited_arxiv_id":null,"evidence_quote":"Supplies the task definition and the standard entity categories (person, organization, location, date, time) that the paper's schema builds on."},{"cited_title":"Enhancement of NER in Low-Resource Languages with Data Augmentation and BERT Models: A Case Study on Urdu","cited_arxiv_id":null,"evidence_quote":"Establishes prior Urdu NER work with data augmentation and BERT, the immediate line of research that EDU-NER-2025 extends into the education domain."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides a deep-learning Urdu NER baseline that the paper positions against and improves upon."},{"cited_title":"Deep recurrent neural networks with word embeddings for Urdu NER","cited_arxiv_id":null,"evidence_quote":"Shows deep recurrent networks with word embeddings for Urdu NER, another prior approach the paper contrasts with transformers."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents the largest existing Urdu NER dataset and 32 embedding-model experiments, the main resource-scarcity benchmark the paper claims to complement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces an earlier Urdu NER corpus and system (KPU-NE with ANN and HMM), giving historical context for the new dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Presents a CRF-based Urdu NER method with the UNER-I dataset, representing the classical supervised approach whose 89% level the transformer surpasses."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Describes a rule-based Urdu NER system, motivating the shift to machine learning and transfer learning that the paper advocates."}],"review_version":1}