Pith. sign in

REVIEW 5 major objections 6 minor 29 references

Named Entity Recognition for Nepali Language

T0 review · 5 major / 6 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read First neural named-entity recognizer for Nepali, built from grapheme-level CNN and bidirectional LSTM, reports test F1 of 86.71 and claims relative gains of 33–50% over a feature-based SVM and up to 10% over prior neural baselines.

desk verdict Useful dataset and first neural NER baseline for Nepali, but the headline numbers don't match the paper's own tables, and the IO tagging plus test-set embedding leakage make the evaluation unreliable. read the letter →

arxiv 1908.05828 v1 pith:6UAHAJF7 submitted 2019-08-16 cs.CL

Oyesh Mann Singh , Ankur Padia , Anupam Joshi This is my paper · ORCID
classification cs.CL
keywords NepaliNERnamedentityrecognitionlow-resourceNLPgraphemeembeddingsBiLSTM-CNNDevanagariscriptfastTextworddatasetcreation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims to build the first neural named-entity recognizer for Nepali, combining a bidirectional LSTM with a CNN that operates on graphemes rather than full characters, and adding one-hot part-of-speech vectors. It also contributes a new labeled news dataset for Nepali and reports test F1 of 86.71 on that dataset and 83.08 on a second annotated dataset. Against the earlier feature-based SVM system for Nepali, the claimed relative improvement is 33–50%, and against the strongest neural models developed for other languages, up to 10%. The authors argue that grapheme-level representations and mild lemmatization of post-positions are particularly effective for a morphologically rich, low-resource language such as Nepali.

What carries the argument

The central object is the grapheme-level CNN feeding a bidirectional LSTM. Graphemes are the atomic units of the Devanagari writing system; for example, a consonant plus a dependent vowel sign forms a single grapheme. The network learns an embedding for each grapheme, applies convolution filters of sizes 3, 4, and 5, max-pools the results, and concatenates this representation with a pre-trained word embedding and a one-hot part-of-speech vector before the BiLSTM layer. This sub-word pathway is what enables the model to capture Nepali's rich morphology without manual feature engineering.

What would settle it

Re-run the experiments on both datasets using the standard BIO/IOB annotation scheme and score all models with that same scheme; if the F1 advantage over the SVM and the neural baselines falls below the claimed 33–50% and 10%, the central claim is contradicted.

Watch

Extended reading notes

Core claim

The central claim is that a BiLSTM-CNN architecture reading words as sequences of graphemes—the atomic writing units of Devanagari—reaches state-of-the-art NER performance for Nepali without hand-crafted features or preprocessing. Using fastText skip-gram embeddings trained on a large Nepali corpus, the model achieves 86.71 F1 on the authors' new OurNepali dataset and 83.08 F1 on a second dataset, beating the strongest neural baselines developed for other languages by up to 10 percent and the earlier SVM-based Nepali NER by 33–50 percent. The paper attributes the gains to two design choices: sub-word awareness via grapheme CNNs, which captures inflectional morphology, and lemmatizing Nepali post-positions, which alone is reported to raise F1 by about 10 points. The authors release the dataset and code to support further work on Nepali and other Devanagari-script languages.

Load-bearing premise

The reported gains depend on the IO tagging scheme, which omits Beginning tags; because a system that only marks the inside of an entity can still be scored correct, the comparison to BIO-based baselines may overstate the true improvements.

Editorial extensions

If this is right

  • The released OurNepali dataset provides a standard benchmark for future Nepali NER research, since no public neural-ready dataset existed before.
  • The grapheme-level CNN can be transferred to other Devanagari-script languages, such as Hindi and Bengali, which share similar writing-system morphology.
  • Lemmatizing post-positions is a cheap, language-specific preprocessing step that the paper shows can improve NER F1 by approximately 10 points in an inflectional language.
  • The BiLSTM + CNN + POS architecture is a plug-and-play recipe that can be applied to other low-resource languages with small annotated corpora.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Re-evaluating with the standard BIO/IOB tagging scheme (which includes Beginning tags) is likely to lower the reported F1 scores; the IO scheme used in the paper can credit a system for tagging the inside of an entity without finding its start, so the exact margins over BIO-trained baselines remain uncertain.
  • The 33–50% improvement over the SVM baseline may shrink if the SVM is supplied with context features; the paper's SVM baseline does not use surrounding words during training.
  • Testing the grapheme-level representation on other scripts, such as Bengali or Gujarati, would clarify whether the gain comes from the writing system or from the particular morphology of Nepali.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper presents the first neural named entity recognition (NER) system for Nepali, built on BiLSTM architectures with CNN-based character- and grapheme-level representations, optionally augmented with POS one-hot vectors. The authors introduce a new Nepali NER dataset (OurNepali), report results on this dataset and on an ILPRL dataset, and compare against a feature-based SVM baseline and published neural baselines. The abstract claims a relative improvement of 33% to 50% over the SVM model and up to 10% over state-of-the-art neural models. The paper also reports that word embeddings were trained on a corpus mixed with the NER dataset texts and that evaluation was performed using an IO tagging scheme without B tags.

Significance. If the claimed results were reproducible, the paper would provide a useful low-resource benchmark: a new Nepali NER dataset, a first neural baseline, and a comparison of embedding and representation choices. The entity-wise F1 tables, the dropout analysis, and the comparison of word2vec/GloVe/FastText variants are potentially informative. However, the central performance claim is internally contradicted by the paper's own tables, and two methodological choices (IO-only evaluation and embedding training mixed with test data) undermine the validity of the reported scores. The dataset contribution is further weakened because the promised repository link is a placeholder.

major comments (5)
  1. [Abstract; Table 6] The headline claim of 'relative improvement of 33% to 50% compared to feature based SVM model' is not supported by the paper's own numbers. On OurNepali, the best model achieves 86.71 versus the SVM baseline at 66.26, a relative gain of (86.71-66.26)/66.26 = 30.9%. On ILPRL, the best model achieves 83.08 versus 46.26, a relative gain of 79.6%. Neither value falls in the advertised 33-50% interval. The 'up to 10% improvement over state-of-the-art neural based model' is also unmatched: relative gains over Lample et al. with fastText are about 1.1% on OurNepali and about 1.0% on ILPRL, and the gain over Lample et al. with word2vec on OurNepali is about 0.25%. The abstract must be corrected or the experiments must be redone.
  2. [Section 5.1; Table 6] Evaluating with an IO tag scheme without B tags is not standard NER evaluation. The CoNLL-2003 evaluation script, which the paper says it uses, is designed for B/I-tagged data and identifies entities as maximal B-I sequences; with only I and O tags, adjacent entities of the same type are merged into one entity, and the model is never penalized for failing to label the beginning of an entity. This inflates F1 scores and makes the comparison with published BIO-labeled baselines (Bam et al., Ma and Hovy, Lample et al.) invalid. The ILPRL dataset reportedly follows the IOB format, so converting it to IO discards exactly the boundary information needed for a fair NER comparison.
  3. [Section 3.2.1; Tables 3 and 6] The word embeddings were trained on a corpus that was 'mixed with the texts from the dataset' before training. Because the dataset includes the test split, the unsupervised embedding training has access to the test sentences, and those sentences also influence the CBOW and skip-gram representations used as model inputs. This is a test-data leakage that inflates the reported F1 scores and breaks comparability with systems using pre-trained embeddings trained on external corpora. The generalization claim of the paper is therefore not supported by the experiments as described.
  4. [Section 4.1; Abstract] The abstract claims the proposed model 'doesn't require any hand-crafted features and no data pre-processing,' but Section 4.1 describes lemmatizing 299 post-positions with a hand-curated list and training a separate BiLSTM POS tagger to label the dataset. Table 3 shows that lemmatizing post-positions 'boosted the F1 score by almost 10%.' This is a substantial, manually supplied preprocessing step, and its effect is not separated from the neural model's contribution. The contradiction between the abstract's claim and the experimental setup should be resolved.
  5. [Section 1; Section 4.1] The paper promises to release its dataset and code, but the link given is 'github.com/link-yet-to-be-updated'. Since a new dataset is one of the paper's stated contributions, the absence of an actual release makes the results non-reproducible. The authors should either provide a working URL or state clearly that the resource is not yet available.
minor comments (6)
  1. [Section 2] There is a spelling error: 'embbeddings' should be 'embeddings'.
  2. [Section 5.3] The learning rate range is written as '(0,1, 0.01, 0.001, 0.0001)'; this appears to be missing a delimiter and should be '0.1, 0.01, 0.001, 0.0001'.
  3. [Section 4.1] 'Since, we there was no publicly available standard Nepali NER dataset' is ungrammatical; consider 'Since there was no publicly available standard Nepali NER dataset'.
  4. [Section 3.2.1] The text says 'shown in 4' when referring to the PCA visualization; it should say 'Figure 4'.
  5. [Section 3.2.3 and Section 4.1] The reference to 'table 3' in Section 3.2.3 for a sample sentence conflicts with Table 3, which is a results table; the sample sentence appears to be in Figure 3.
  6. [Section 7] 'highly inflectional like Nepali' should be 'highly inflectional languages like Nepali'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; the paper's empirical claims are compromised by evaluation validity issues, but those are not circularity.

full rationale

The paper makes empirical claims about a new dataset and neural architecture for Nepali NER; there is no formal derivation whose conclusion is equivalent to its own inputs. No equation in the paper defines a predicted quantity in terms of the same quantity, and no fitted parameter is renamed as a prediction. The paper contains no load-bearing self-citations and invokes no uniqueness theorem from the authors' prior work. I flag two validity concerns that are not circularity: (1) Section 5.1 states the models are trained on IO format 'hence the dataset does not contain any B-type annotation unlike in BIO,' which makes direct comparison with published BIO-based baselines questionable; (2) Section 3.2.1 says the word-embedding corpus 'was mixed with the texts from the dataset before training CBOW and skip-gram version of word2vec,' so test-set sentences contribute to the learned embeddings, a test-set leakage that inflates reported F1. In addition, the abstract's headline improvements are not reproducible from Table 6: the best F1 against the SVM baseline is about 30.9% on OurNepali and 79.6% on ILPRL, not the stated 33–50%, and the neural gains over Lample et al. are about 1%, not 10%. These are internal-inconsistency and evaluation-validity problems, not cases of an output reducing to an input by construction. Therefore no circular step can be exhibited, and the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim rests on a newly created dataset, a reimplementation of baselines, and several corpus and tagging choices that are not independently verified.

free parameters (5)
  • LSTM hidden size = 100
    Selected by hyperparameter search (Section 5.3, Table 4).
  • Dropout rate = 0.5
    Chosen from dropout sweep shown in Figure 5.
  • CNN filter sizes and count = [3,4,5], 30 filters
    Chosen after ablation studies (Section 5).
  • Character/grapheme embedding dimension = 30
    Set in Section 3.2.2, not justified by data.
  • FastText subword size range = 1 to 4
    Chosen by the authors for Nepali morphology (Section 3.2.1).
assumptions (5)
  • domain assumption The ILPRL dataset annotations are usable after unspecified 'few corrections'
    Section 4.2 states corrections were made without describing them or measuring inter-annotator agreement.
  • domain assumption The POS-tagger trained on Nepali National Corpus (95.14% accuracy) gives reliable labels for NER training
    Section 4.1 uses it to tag the dataset.
  • ad hoc to paper Lemmatizing a hand-curated list of 299 post-positions is a valid normalization
    Section 4.1, chosen because it boosted F1 by 10% on the dev set.
  • ad hoc to paper FastText subword sizes 1-4 capture Nepali morphology
    Section 3.2.1.
  • ad hoc to paper IO tagging without B tags is a valid evaluation for NER
    Section 5.1, necessary for their comparisons but not standard.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Named Entity Recognition for Nepali Language." pith.science (2026). https://pith.science/paper/6UAHAJF7

@misc{pith2026190805828,
  author       = {Pith},
  title        = {Pith review of: Named Entity Recognition for Nepali Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6UAHAJF7}},
  note         = {Machine review of arXiv:1908.05828}
}
read the original abstract

Named Entity Recognition have been studied for different languages like English, German, Spanish and many others but no study have focused on Nepali language. In this paper we propose a neural based Nepali NER using latest state-of-the-art architecture based on grapheme-level which doesn't require any hand-crafted features and no data pre-processing. Our novel neural based model gained relative improvement of 33% to 50% compared to feature based SVM model and up to 10% improvement over state-of-the-art neural based model developed for languages beside Nepali.

Figures

Figures reproduced from arXiv: 1908.05828 by the authors.

Figure 1
Figure 1. The grapheme level convolution neural net [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. End-to-end model architecture of our neural network. W-EMB, GRAPH, POS represents pre-trained [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Format of a sample sentence in our dataset. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: 2D Visualization of nearest neighbor word using PCA for a sample word [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: F1 score based on different dropout val [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Sample output of the best model from ILPRL [PITH_FULL_IMAGE:figures/full_fig_p007_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

29 extracted references · 9 canonical work pages

  1. [1]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Alan Akbik, Duncan Blythe, and Roland Vollgraf. 2018. https://www.aclweb.org/anthology/C18-1139 Contextual string embeddings for sequence labeling . In Proceedings of the 27th International Conference on Computational Linguistics, pages 1638--1649, Santa Fe, New Mexico, USA. Association for Computational Linguistics

    2018

  4. [4]

    Vinayak Athavale, Shreenivas Bharadwaj, Monik Pamecha, Ameya Prabhu, and Manish Shrivastava. 2016. http://arxiv.org/abs/1610.09756 Towards deep learning in hindi NER: an approach to tackle the labelled data sparsity . CoRR, abs/1610.09756

    work page Pith review arXiv 2016

  5. [5]

    Surya Bahadur Bam and Tej Bahadur Shahi. 2014. Named entity recognition for nepali text using support vector machines. Intelligent Information Management, 6(02):21

    work page 2014

  6. [6]

    Nayan Banik and Md Hasan Hafizur Rahman. 2018. Gru based named entity recognition system for bangla online newspapers. In 2018 International Conference on Innovation in Engineering and Technology (ICIET), pages 1--6. IEEE

    work page 2018

  7. [7]

    Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2016. Enriching word vectors with subword information. arXiv preprint arXiv:1607.04606

    arXiv 2016

  8. [8]

    Jason P. C. Chiu and Eric Nichols. 2015. http://arxiv.org/abs/1511.08308 Named entity recognition with bidirectional lstm-cnns . CoRR, abs/1511.08308

    arXiv 2015

Show all 29 references
  1. [9]

    Junyoung Chung, C aglar G \" u l c ehre, KyungHyun Cho, and Yoshua Bengio. 2014. http://arxiv.org/abs/1412.3555 Empirical evaluation of gated recurrent neural networks on sequence modeling . CoRR, abs/1412.3555

    2014 arXiv

  2. [10]

    Ronan Collobert, Jason Weston, L \' e on Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel P. Kuksa. 2011. http://arxiv.org/abs/1103.0398 Natural language processing (almost) from scratch . CoRR, abs/1103.0398

    2011 arXiv

  3. [11]

    Jacob Devlin, Ming - Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. http://arxiv.org/abs/1810.04805 BERT: pre-training of deep bidirectional transformers for language understanding . CoRR, abs/1810.04805

    2018 arXiv

  4. [12]

    Arindam Dey, Abhijit Paul, and Bipul Syam Purkayastha. 2014. Named entity recognition for nepali language: A semi hybrid approach. International Journal of Engineering and Innovative Technology (IJEIT) Volume, 3:21--25

    2014

  5. [13]

    Arindam Dey and Bipul Syam Prukayastha. 2013. Named entity recognition using gazetteer method and n-gram technique for an inflectional language: A hybrid approach. International Journal of Computer Applications, 84(9)

    2013

  6. [14]

    Chris Dyer, Miguel Ballesteros, Wang Ling, Austin Matthews, and Noah A. Smith. 2015. http://arxiv.org/abs/1505.08075 Transition-based dependency parsing with stack long short-term memory . CoRR, abs/1505.08075

    2015 arXiv

  7. [15]

    Jenny Rose Finkel, Trond Grenager, and Christopher Manning. 2005. Incorporating non-local information into information extraction systems by gibbs sampling. In Proceedings of the 43rd annual meeting on association for computational linguistics, pages 363--370. Association for ...

    2005

  8. [16]

    William Gunawan, Derwin Suhartono, Fredy Purnomo, and Andrew Ongko. 2018. Named-entity recognition for indonesian language using bidirectional lstm-cnns. Procedia Computer Science, 135:425--432

    2018

  9. [17]

    Ayush Gupta, Meghna Ayyar, Ashutosh Kumar Singh, and Rajiv Ratn Shah. 2018. raiden11@ iecsil-fire-2018: Named entity recognition for indian languages. FIRE Working Notes

    2018

  10. [18]

    Sepp Hochreiter and J \"u rgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735--1780

    1997

  11. [19]

    Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer. 2016. http://arxiv.org/abs/1603.01360 Neural architectures for named entity recognition . CoRR, abs/1603.01360

    2016 arXiv

  12. [20]

    Yann LeCun, Bernhard Boser, John S Denker, Donnie Henderson, Richard E Howard, Wayne Hubbard, and Lawrence D Jackel. 1989. Backpropagation applied to handwritten zip code recognition. Neural computation, 1(4):541--551

    1989

  13. [21]

    Xuezhe Ma and Eduard H. Hovy. 2016. http://arxiv.org/abs/1603.01354 End-to-end sequence labeling via bi-directional lstm-cnns-crf . CoRR, abs/1603.01354

    2016 arXiv

  14. [22]

    Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. http://arxiv.org/abs/1310.4546 Distributed representations of words and phrases and their compositionality . CoRR, abs/1310.4546

    2013 arXiv

  15. [23]

    Suzi Park and Hyopil Shin. 2018. https://www.aclweb.org/anthology/L18-1471 Grapheme-level awareness in word embeddings for morphologically rich languages . In Proceedings of the 11th Language Resources and Evaluation Conference, Miyazaki, Japan. European Language Resource Association

    2018

  16. [24]

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. 2017. Automatic differentiation in pytorch

    2017

  17. [25]

    Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. https://doi.org/10.3115/v1/D14-1162 G love: Global vectors for word representation . In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing ( EMNLP ) , pages 1532--1543, Doha,...

    2014 doi

  18. [26]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. http://arxiv.org/abs/1802.05365 Deep contextualized word representations . CoRR, abs/1802.05365

    2018 arXiv

  19. [27]

    Radim R eh u r ek and Petr Sojka. 2010. Software Framework for Topic Modelling with Large Corpora . In Proceedings of the LREC 2010 Workshop on New Challenges for NLP Frameworks , pages 45--50, Valletta, Malta. ELRA. http://is.muni.cz/publication/884893/en

    2010

  20. [28]

    Rumelhart, Geoffrey E

    David E. Rumelhart, Geoffrey E. Hinton, and Ronald J. Williams. 1988. http://dl.acm.org/citation.cfm?id=65669.104451 Neurocomputing: Foundations of research . chapter Learning Representations by Back-propagating Errors, pages 696--699. MIT Press, Cambridge, MA, USA

    1988

  21. [29]

    Tjong Kim Sang and Fien De Meulder

    Erik F. Tjong Kim Sang and Fien De Meulder. 2003. https://doi.org/10.3115/1119176.1119195 Introduction to the conll-2003 shared task: Language-independent named entity recognition . In Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003 - Volum...

    2003

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.