NepaliGPT: A Generative Language Model for the Nepali Language

Aman Shakya; Sahil Bhatta; Shushanta Pudasaini; Siddhartha Shrestha; Sunil Thapa; Sushmita Palikhe

REVIEW 5 major objections 6 minor 13 references

NepaliGPT: A Generative Language Model for the Nepali Language

T0 review · 5 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper proposes NepaliGPT, a from-scratch GPT-2-style causal language model for Nepali, reporting 26.32 perplexity and adding a 9.3 GB Devanagari corpus and a 4,296-pair QA benchmark.

desk verdict A useful artifact in an under-resourced language, but the reported metrics are internally inconsistent and the 'first generative Nepali LM' claim is asserted, not demonstrated; worth a referee's time only if the authors can fix the eval and release the model. read the letter →

arxiv 2506.16399 v1 pith:PJDKW4ML submitted 2025-06-19 cs.CL cs.AI

Shushanta Pudasaini , Aman Shakya , Siddhartha Shrestha , Sahil Bhatta , Sunil Thapa , Sushmita Palikhe This is my paper · ORCID

classification cs.CLcs.AI

keywords NepalilanguagegenerativemodelcausalmodelingDevanagariscriptbyte-pairencodingperplexityquestionansweringlow-resourceNLP

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that no generative language model exists for Nepali, and that the gap can be filled with a comparatively small, from-scratch causal model. The authors build NepaliGPT by training a GPT-2-style transformer on the Devanagari Corpus, about 383 million tokens of scraped and translated Nepali text, then fine-tuning it on 4,296 Nepali question-answer pairs. They report 26.32 perplexity, ROUGE-1 around 0.26, and human ratings of roughly 81% causal coherence and 85% causal consistency, which they read as evidence that a native Nepali generative model is viable. A sympathetic reader would care because the claim, if true, gives Nepali speakers a generative model that produces Devanagari text directly rather than routing through English.

What carries the argument

The carrying mechanism is a causal language model: a transformer decoder that learns to predict the next token given the previous ones, trained from a randomly initialised GPT-2-style checkpoint on a custom byte-pair-encoding tokeniser over Devanagari script. The tokeniser, built to a vocabulary of about 10,000 subwords, keeps the vocabulary manageable for the script's large alphabet; pretraining on the 9.3 GB corpus supplies language knowledge; full fine-tuning on question-answer pairs turns that knowledge into a usable Nepali question-answering system.

What would settle it

Compile a release-dated list of generative language models that can produce Nepali text and check whether any was publicly available before 19 June 2025; the existence of even one such model falsifies the paper's novelty claim. A separate check: recompute ROUGE-1 on the released test set, since the paper reports 0.2604 in the abstract and 0.1501 in the conclusion, two values that cannot both be the same metric.

Watch

Extended reading notes

Core claim

The paper's central claim is that a generative language model for Nepali is both missing and buildable, and that NepaliGPT is that model. On its own terms, the discovery is that a GPT-2-style causal language model trained from scratch on roughly 383 million Nepali tokens reaches a perplexity of 26.32, close to the 24.35 the authors cite for GPT-2 on English Wikipedia, while its fine-tuned question-answering variant receives human ratings of 81.25% causal coherence and 85.41% causal consistency. The same research introduces the Devanagari Corpus (about 9.3 GB) and a Nepali question-answer benchmark of 4,296 pairs, which the authors present as reusable resources so that later work does not have to start from raw scraped text.

Load-bearing premise

The load-bearing premise is that no other generative language model for Nepali exists; the paper supports this only by mentioning two BERT-style models, so if any Nepali-capable generative model predates this work, the novelty claim collapses.

Editorial extensions

If this is right

If the results hold, Nepali can support native generative applications—chatbots, article drafting, and question answering—without an English translation step.
The Devanagari Corpus gives future work a ready-made pretraining resource, so later Nepali models do not have to scrape and clean text from scratch.
The 4,296-pair question-answer benchmark gives the field a fixed test set for comparing Nepali generative models.
Fine-tuning NepaliGPT on instruction-style data would plausibly produce an instruction-following Nepali assistant, as the paper itself suggests.
Bigger, more diverse corpora should lower perplexity further, since the larger of the two pretraining experiments already improves perplexity from 87 to 26.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If the same recipe transfers, other Devanagari-script or low-resource languages could get a GPT-style model from a few gigabytes of scraped text, a byte-pair tokeniser, and a modest GPU.
The paper's claim that no generative Nepali model exists is not established by its two-model literature review; a systematic sweep of model repositories would be the natural check, and its absence leaves the novelty claim hostage to timing.
The reported performance suggests that language-specific models do not necessarily need massive scale, which undercuts the assumption that low-resource languages must wait for large multilingual models.
A direct head-to-head against a fine-tuned multilingual model on the same Nepali QA benchmark would make the native-model advantage testable; the paper does not run that comparison.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a straightforward engineering report—train a GPT-2-style CLM on a scraped Nepali corpus, add a QA fine-tune dataset, report perplexity and a few human-judged scores. The artifact itself is genuinely new and useful for a language with thin NLP resources: a 9.3 GB Devanagari corpus, a 4,296-pair QA dataset, and a trained Nepali tokenizer are things people in the low-resource community would actually use. The paper's claim to be the first generative Nepali LM may well be right, but it's asserted rather than surveyed; the literature review lists only two BERT-style encoders and never shows a systematic search. That's a real weakness but a fixable one.

The bigger problem is internal: the abstract and Table 8 report ROUGE-1 as 0.2604, while the same table and the conclusion report 0.1501. Both are labeled ROUGE-1 for the same 50-sample evaluation, with no caveat about a different setting. Whatever the truth, the paper as written does not establish the headline performance claim. The evaluation is also thin in other ways: no variance or significance, a nonstandard half-sentence ROUGE protocol for the CLM, three annotators with no agreement measure, and no released model or code. None of these are fatal on their own—this is a resource paper, not a theoretical contribution—but the metric contradiction is load-bearing because it undermines the central quantitative claim. The self-citation pattern is fine: citing your own earlier NepaliBERT is appropriate here, though the novelty argument would be stronger with an actual search for concurrent generative Nepali models.

Where the paper stands after that: the data-collection and preprocessing details are concrete and reproducible in principle, the training curves look plausible, and the perplexity (26.32) is in a reasonable range for a small GPT-2 on a low-resource language. But without a public model, a corrected metric table, and at least one independent baseline, I would not rely on the reported numbers. For a serious referee, the paper belongs on a borderline: the artifact deserves attention, but the manuscript needs a revision that fixes the contradiction and releases the model. I'd rather see this resubmitted with those fixes than accepted as-is.

For peer review: yes, send it to referees—the resource value is real and the community needs to know what's available. For my own work, I'd cite it for the corpus and dataset, not for the model's claimed quality.

Referee Report

5 major / 6 minor

Summary. The paper reports the development of NepaliGPT, a causal language model based on the GPT-2 architecture and trained from scratch on a newly collected Nepali corpus named the Devanagari Corpus. The authors describe a custom BPE tokenizer, pre-training on a large Nepali text corpus, fine-tuning on a Nepali question-answering dataset of 4,296 pairs, and an evaluation that reports perplexity of 26.32245, ROUGE-1 of 0.2604, causal coherence of 81.25%, and causal consistency of 85.41%. The paper claims that NepaliGPT is the first generative language model for Nepali, with the corpus and QA benchmark offered as reusable resources. The central contribution is therefore an empirical training and evaluation report for a low-resource language model.

Significance. If the claims are substantiated, the paper would provide a valuable Nepali language resource: a causal LM, a large raw corpus, a custom tokenizer, and a QA benchmark for a language with relatively few NLP resources. The paper also makes the falsifiable claim that no prior generative LM exists for Nepali, which would sharpen the novelty. However, the evaluation as presented does not currently support the headline numbers: the ROUGE values are internally contradictory, the evaluation protocol is nonstandard and lacks baselines or variance estimates, and the human evaluation lacks essential reporting details. The paper also does not release code, model weights, or a public dataset link, limiting reproducibility. These issues are addressable in a revision, and the underlying training pipeline appears plausible, so the contribution has potential if the evaluation is substantially strengthened and the inconsistencies corrected.

major comments (5)

[Abstract; Table 8; Section 6] The central performance metric is internally contradictory. The abstract reports a ROUGE-1 score of 0.2604, but Table 8 lists both ROGUE-1 = 0.2604 and ROGUE-1 = 0.1501 for the same NepaliGPT model on the same held-out evaluation, and Section 6 states that the model received a ROUGE-1 score of 0.1501. No explanation is given for the two different ROUGE-1 values, such as different evaluation settings, different reference texts, or different data subsets. Because this metric is a core piece of evidence that the model can generate meaningful Nepali text, the inconsistency must be resolved before the results can be interpreted.
[Section 4.3.1; Section 5.2; Table 8] The ROUGE evaluation protocol is nonstandard and underpowered. The authors removed the second half of 50 sentences, treated the removed part as reference text, and computed ROUGE between the model's predicted continuation and that reference. This is not the summarization-style annotation protocol for which ROUGE is defined, no metric implementation is cited, no confidence intervals or standard deviations are given, and no baseline is evaluated under the same protocol. With only 50 examples and no comparison point, the numbers 0.2604 and 0.1501 cannot be interpreted as evidence about generation quality.
[Section 4.3.2; Table 5; Table 9] The causal coherence and causal consistency scores are not adequately documented. The paper reports 81.25% coherence and 85.41% consistency from three annotators but does not state how many generated outputs were evaluated, what instructions the annotators received, how disagreements were resolved, or what the inter-annotator agreement was. Given the small annotator pool and the complete absence of example outputs in the paper, these percentages are not sufficient to support the claim that the model produces causally coherent and consistent text.
[Section 3.1.2, Table 1; Section 3.2.2; Table 3] There are serious inconsistencies in the reported corpus size and training configuration. Table 1 reports 383,303,233 tokens and a 9.3 GB corpus, while Section 3.2.2 refers to a '13 million Nepali text corpus' and Table 3 lists the corpus size as 9.6 GB. The paper also says the GPT-2 architecture was initialized with 'zero weight initialisation,' which is not standard and is not further explained, and no model dimensions (number of layers, hidden size, attention heads, parameter count) are given. These inconsistencies prevent the reader from reproducing or even understanding what was trained.
[Section 1; Section 2] The novelty claim that 'there isn't any generative language model for the Nepali language' is asserted without a systematic literature search. The literature review cites only NepaliBERT and NepBERTa, both BERT-style understanding models, but does not discuss generative models, fine-tuned multilingual LLMs, or concurrent Nepali GPT variants. If such a model exists, the paper's framing as the 'first' generative Nepali LM would need to be withdrawn, though the corpus and QA benchmark could still stand as contributions. The authors should either provide a more thorough survey or soften the claim.

minor comments (6)

[Throughout] The metric name is consistently misspelled as 'ROGUE' in Section 4.3.1 and Table 8; the correct term is ROUGE.
[Section 4.3.2] The headings 'Casual Coherence' and 'Casual Consistency' should be 'Causal Coherence' and 'Causal Consistency.'
[Section 3.1.1] The filtering criterion 'checking if any alphanumeric character existed' is not a meaningful cleaning step, since nearly all Nepali text contains alphanumeric characters; the intended preprocessing should be described precisely.
[Section 5.3] The paper mentions inference time recorded from HuggingFace API endpoints but gives no model identifier or repository link; adding a public model card would substantially improve reproducibility.
[Table 8] The column 'Benchmark for English Dataset' is unexplained, and no source is given for the GPT-2 ROUGE values; the comparison in Section 6 to GPT-2's ROUGE-1 of 0.148 therefore lacks a verifiable reference.
[Section 5.1] There is an unfinished phrase, 'othe utput of the Two of the experiments,' which should be corrected to 'the output of two of the experiments.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; empirical training report with measured metrics, not a derivation that reduces to its own inputs.

full rationale

This paper is an empirical training report. NepaliGPT is a GPT-2-style causal language model trained from scratch on a newly assembled Nepali corpus and then fine-tuned on question-answer pairs. The reported outcomes—perplexity, ROUGE scores, and human-rated causal coherence/consistency—are measurements taken on held-out or unseen evaluation data, not quantities derived from the model's own training objective by construction. There is no equation-level reduction in which an input parameter is renamed as a prediction. The only self-citation is the authors' earlier NepaliBERT work (reference [6]) in the literature review, and it is used merely to describe the existing NLU-style models, not to justify the architecture, training procedure, or any quantitative claim. The assertion that no generative Nepali language model exists is an unverified external novelty claim rather than a circular step; if wrong, it weakens the contribution but does not make the reported measurements tautological. The internal inconsistency between the ROUGE-1 values 0.2604 and 0.1501 in Table 8 and the Conclusion is a serious correctness and reproducibility problem, but it is not circularity: both values are reported as empirical evaluation results, not as fitted inputs used to generate the claim. Because the derivation chain is empirical and self-contained, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces a named corpus (Devanagari Corpus) and a QA dataset, but both are only described, not released, and their construction relies on unquantified assumptions about translation quality and source representativeness. No new theoretical objects are postulated.

assumptions (3)

domain assumption Google Translate from English to Nepali yields adequate training text for fine-tuning.
Section 3.3.2: 4000 English QA pairs are translated via the Google Translate API and used for fine-tuning; no quality metric beyond manual spot checks is given.
domain assumption Scraped news portals and open NLP datasets form a representative sample of Nepali.
Section 3.1.1 lists ekantipur.com, onlinekhabar.com, OSCAR, and the Nepali National Corpus, but no domain-bias analysis is provided.
ad hoc to paper Perplexity and ROUGE measured on 50 held-out examples reliably estimate model quality.
Section 5.2: only 50 unseen data points are used, with no confidence intervals or repeated-seed variance reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of NepaliGPT: A Generative Language Model for the Nepali Language." pith.science (2026). https://pith.science/paper/PJDKW4ML

@misc{pith2026250616399,
  author       = {Pith},
  title        = {Pith review of: NepaliGPT: A Generative Language Model for the Nepali Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PJDKW4ML}},
  note         = {Machine review of arXiv:2506.16399}
}

read the original abstract

After the release of ChatGPT, Large Language Models (LLMs) have gained huge popularity in recent days and thousands of variants of LLMs have been released. However, there is no generative language model for the Nepali language, due to which other downstream tasks, including fine-tuning, have not been explored yet. To fill this research gap in the Nepali NLP space, this research proposes \textit{NepaliGPT}, a generative large language model tailored specifically for the Nepali language. This research introduces an advanced corpus for the Nepali language collected from several sources, called the Devanagari Corpus. Likewise, the research introduces the first NepaliGPT benchmark dataset comprised of 4,296 question-answer pairs in the Nepali language. The proposed LLM NepaliGPT achieves the following metrics in text generation: Perplexity of 26.32245, ROUGE-1 score of 0.2604, causal coherence of 81.25\%, and causal consistency of 85.41\%.

Figures

Figures reproduced from arXiv: 2506.16399 by the authors.

**Figure 2.** Wordcloud representation of the Devanagari Corpus [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Bar graph of the most frequent words in the Devanagari Corpus [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (6 more)

**Figure 4.** Figure 4: Block Diagram representing methodology for training the custom Nepali BPE Tokeniser [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Data format used for Question Answering Task [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]

**Figure 6.** Figure 6: Finetuning Nepali Causal Language Model 4 Experimental Setup 4.1 Hyperparameters used for Pretraining Nepali CLM We ran multiple experiments for building Nepali CLM i.e. one with a smaller data corpus and the other with a bigger data corpus. For hyperparameters on both…

**Figure 7.** Figure 7: Data Samples Used for Evaluation 4.3.2 Casual Coherence and Casual Consistency To measure these metrics, we used three different human evaluators, each with a different profile. The profile of the evaluators is as mentioned in the [PITH_FULL_IMAGE:figures/full_fig_p00…

**Figure 8.** Figure 8: Epoch Vs Perplexity from experiment I For the second experiment, i.e. with a bigger data corpus of 9.6 GB following were the obtained training metrics: Epoch Training Loss Validation Loss Perplexity 1.0 3.7654 3.6291 37.6789 2.0 3.5857 3.4442 31.31812 3.0 3.5051 3.3439…

**Figure 9.** Figure 9: Epoch Vs Perplexity from experiment II 5.2 NepaliGPT We selected 50 unseen data points from our test dataset and calculated the Rogue-N score and causal coherence, and consistency. The complete metrics are shown below. ROUGE Type Score Validation Loss Benchmark for Eng…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

13 extracted references · 9 canonical work pages

[1]

Rajesh Khatiwada. Nepali. Journal of the International Phonetic Association, 39(3):373–380, 2009

work page 2009
[2]

Linguistic geography of nepalese languages

Rajendra Khanal. Linguistic geography of nepalese languages. The Third Pole: Journal of Geography Education, pages 45–54, 2019

work page 2019
[3]

Survey of nlp resources in low-resource languages nepali, sindhi and konkani

Annie Rajan and Ambuja Salgaonkar. Survey of nlp resources in low-resource languages nepali, sindhi and konkani. In Amit Joshi, Mufti Mahmud, Roshan G. Ragel, and Nileshsingh V . Thakur, editors,Information and Communication Technology for Competitive Strategies (ICTCS 2020), pages 121–132, Singapore, 2022. Springer Singapore

work page 2020
[4]

Vector representation based on a supervised codebook for nepali documents classification

Chiranjibi Sitaula, Anish Basnet, and Sunil Aryal. Vector representation based on a supervised codebook for nepali documents classification. PeerJ Computer Science, 7:e412, 2021

work page 2021
[5]

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

2017
[6]

Nepalibert: Pre-training of masked language model in nepali corpus

Shushanta Pudasaini, Subarna Shakya, Aakash Tamang, Sajjan Adhikari, Sunil Thapa, and Sagar Lamichhane. Nepalibert: Pre-training of masked language model in nepali corpus. In 2023 7th International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud)(I-SMAC), pages 325–330. IEEE, 2023

work page 2023
[7]

Nepberta: Nepali language model trained in a large corpus

Sulav Timilsina, Milan Gautam, and Binod Bhattarai. Nepberta: Nepali language model trained in a large corpus. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 273–284, 2022

work page 2022
[8]

Construction and annotation of a corpus of contemporary nepali

Yogendra P Yadava, Andrew Hardie, Ram Raj Lohani, Bhim N Regmi, Srishtee Gurung, Amar Gurung, Tony McEnery, Jens Allwood, and Pat Hall. Construction and annotation of a corpus of contemporary nepali. Corpora, 3(2):213–225, 2008

work page 2008

Show all 13 references

[9]

A large scale nepali text corpus

Rabindra Lamsal. A large scale nepali text corpus. IEEE Dataport, 2020

2020
[10]

A monolingual approach to contextualized word embeddings for mid-resource languages

Pedro Javier Ortiz Suárez, Laurent Romary, and Benoît Sagot. A monolingual approach to contextualized word embeddings for mid-resource languages. arXiv preprint arXiv:2006.06202, 2020

2006 arXiv
[11]

Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing

Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. arXiv preprint arXiv:1808.06226, 2018

2018 arXiv
[12]

A new algorithm for data compression

Philip Gage. A new algorithm for data compression. The C Users Journal, 12(2):23–38, 1994

1994
[13]

Language models are unsupervised multitask learners

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9, 2019. 11

2019

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

Rajesh Khatiwada. Nepali. Journal of the International Phonetic Association, 39(3):373–380, 2009

work page 2009

[2] [2]

Linguistic geography of nepalese languages

Rajendra Khanal. Linguistic geography of nepalese languages. The Third Pole: Journal of Geography Education, pages 45–54, 2019

work page 2019

[3] [3]

Survey of nlp resources in low-resource languages nepali, sindhi and konkani

Annie Rajan and Ambuja Salgaonkar. Survey of nlp resources in low-resource languages nepali, sindhi and konkani. In Amit Joshi, Mufti Mahmud, Roshan G. Ragel, and Nileshsingh V . Thakur, editors,Information and Communication Technology for Competitive Strategies (ICTCS 2020), pages 121–132, Singapore, 2022. Springer Singapore

work page 2020

[4] [4]

Vector representation based on a supervised codebook for nepali documents classification

Chiranjibi Sitaula, Anish Basnet, and Sunil Aryal. Vector representation based on a supervised codebook for nepali documents classification. PeerJ Computer Science, 7:e412, 2021

work page 2021

[5] [5]

Attention is all you need

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

2017

[6] [6]

Nepalibert: Pre-training of masked language model in nepali corpus

Shushanta Pudasaini, Subarna Shakya, Aakash Tamang, Sajjan Adhikari, Sunil Thapa, and Sagar Lamichhane. Nepalibert: Pre-training of masked language model in nepali corpus. In 2023 7th International Conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud)(I-SMAC), pages 325–330. IEEE, 2023

work page 2023

[7] [7]

Nepberta: Nepali language model trained in a large corpus

Sulav Timilsina, Milan Gautam, and Binod Bhattarai. Nepberta: Nepali language model trained in a large corpus. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 273–284, 2022

work page 2022

[8] [8]

Construction and annotation of a corpus of contemporary nepali

Yogendra P Yadava, Andrew Hardie, Ram Raj Lohani, Bhim N Regmi, Srishtee Gurung, Amar Gurung, Tony McEnery, Jens Allwood, and Pat Hall. Construction and annotation of a corpus of contemporary nepali. Corpora, 3(2):213–225, 2008

work page 2008