{"id":"55448dc4-d8e0-4a53-8eca-bb943d2aef26","arxiv_id":"2506.16399","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper describes training and fine-tuning a GPT-2 language model on Nepali text, reporting perplexity 26.3, but releases no artifacts and presents weakly validated metrics.","lead":"Using a new 9.3 GB Nepali corpus and a small translated question-answer set, the authors train a GPT-2 style model for Nepali and report perplexity of 26.3 with high human-judged coherence. The work is an early resource-building step for low-resource Nepali NLP, but it releases no code, data, or model and has inconsistent evaluation numbers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's reported metrics are internally inconsistent: ROUGE-1 is listed as both 0.2604 and 0.1501, so the central quantitative claim is not established; this is a more direct weakness than the unverified novelty premise.","rationale":"The reader's REJECT verdict is well supported. The paper contains a glaring internal inconsistency: Table 8 reports ROUGE-1 as both 0.2604 and 0.1501, and the conclusion uses 0.1501 while the abstract uses 0.2604. This directly undermines the reported performance metrics, which are central to the claim that NepaliGPT is a functioning generative model. The reader's stated weakest assumption focused on the unverified novelty claim that no generative Nepali LM exists; however, the reader's rationale also mentioned the ROUGE-1 contradiction. I consider the metric contradiction the more load-bearing concern because it is an observable error in the manuscript itself, not dependent on external literature. If the metrics are unreliable, the central empirical claim collapses. The novelty issue is also serious but would only weaken the contribution's framing rather than invalidate the model's existence and utility. Therefore, the verdict remains REJECT. My agreement with the reader is partial: we both identified the metric inconsistency, but I would place it above the novelty premise as the primary reason for rejection.","tokens_in":6069,"tokens_out":7602,"duration_ms":75126,"concrete_test":"Obtain the evaluation script and the 50 held-out question-answer test samples from the authors (or reconstruct them from the described dataset construction). Re-run the ROUGE computation using the exact code and confirm whether the ROUGE-1 score is 0.2604 or 0.1501. If both values are produced by different evaluation settings (e.g., different maximum output lengths or reference truncation), require the paper to state the setting that corresponds to the abstract's headline number. If only one value is reproducible, the other is erroneous and the reported ROUGE-1 in the abstract and conclusion is not trustworthy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that NepaliGPT achieves a ROUGE-1 score of 0.2604 (abstract, Table 8) is contradicted by the same paper: Table 8 also lists a ROUGE-1 score of 0.1501, and the Conclusion states 'the model received a ROUGE-1 score of 0.1501.' Both are labeled ROUGE-1 and appear in the evaluation of NepaliGPT on the 50 held-out samples; no distinction is made between different settings or reference texts. Because the reported metrics are a core part of the claim that NepaliGPT is a working generative model, this internal contradiction makes the quantitative evaluation unreliable. The novelty premise (no prior Nepali generative LM) is also unverified, but the metric contradiction is a demonstrable error within the manuscript itself and does not depend on external literature. Without a released model or code, the reader cannot resolve which ROUGE-1 value is correct, so the central performance claim is unsupported as written. This is the most load-bearing weakness because it directly affects the empirical evidence for the model's capabilities, whereas the novelty premise would only reduce the contribution without invalidating the model's existence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports the development of NepaliGPT, a causal language model based on the GPT-2 architecture and trained from scratch on a newly collected Nepali corpus named the Devanagari Corpus. The authors describe a custom BPE tokenizer, pre-training on a large Nepali text corpus, fine-tuning on a Nepali question-answering dataset of 4,296 pairs, and an evaluation that reports perplexity of 26.32245, ROUGE-1 of 0.2604, causal coherence of 81.25%, and causal consistency of 85.41%. The paper claims that NepaliGPT is the first generative language model for Nepali, with the corpus and QA benchmark offered as reusable resources. The central contribution is therefore an empirical training and evaluation report for a low-resource language model.","tokens_in":6450,"tokens_out":4201,"duration_ms":40870,"significance":"If the claims are substantiated, the paper would provide a valuable Nepali language resource: a causal LM, a large raw corpus, a custom tokenizer, and a QA benchmark for a language with relatively few NLP resources. The paper also makes the falsifiable claim that no prior generative LM exists for Nepali, which would sharpen the novelty. However, the evaluation as presented does not currently support the headline numbers: the ROUGE values are internally contradictory, the evaluation protocol is nonstandard and lacks baselines or variance estimates, and the human evaluation lacks essential reporting details. The paper also does not release code, model weights, or a public dataset link, limiting reproducibility. These issues are addressable in a revision, and the underlying training pipeline appears plausible, so the contribution has potential if the evaluation is substantially strengthened and the inconsistencies corrected.","major_comments":[{"comment":"The central performance metric is internally contradictory. The abstract reports a ROUGE-1 score of 0.2604, but Table 8 lists both ROGUE-1 = 0.2604 and ROGUE-1 = 0.1501 for the same NepaliGPT model on the same held-out evaluation, and Section 6 states that the model received a ROUGE-1 score of 0.1501. No explanation is given for the two different ROUGE-1 values, such as different evaluation settings, different reference texts, or different data subsets. Because this metric is a core piece of evidence that the model can generate meaningful Nepali text, the inconsistency must be resolved before the results can be interpreted.","section":"Abstract; Table 8; Section 6"},{"comment":"The ROUGE evaluation protocol is nonstandard and underpowered. The authors removed the second half of 50 sentences, treated the removed part as reference text, and computed ROUGE between the model's predicted continuation and that reference. This is not the summarization-style annotation protocol for which ROUGE is defined, no metric implementation is cited, no confidence intervals or standard deviations are given, and no baseline is evaluated under the same protocol. With only 50 examples and no comparison point, the numbers 0.2604 and 0.1501 cannot be interpreted as evidence about generation quality.","section":"Section 4.3.1; Section 5.2; Table 8"},{"comment":"The causal coherence and causal consistency scores are not adequately documented. The paper reports 81.25% coherence and 85.41% consistency from three annotators but does not state how many generated outputs were evaluated, what instructions the annotators received, how disagreements were resolved, or what the inter-annotator agreement was. Given the small annotator pool and the complete absence of example outputs in the paper, these percentages are not sufficient to support the claim that the model produces causally coherent and consistent text.","section":"Section 4.3.2; Table 5; Table 9"},{"comment":"There are serious inconsistencies in the reported corpus size and training configuration. Table 1 reports 383,303,233 tokens and a 9.3 GB corpus, while Section 3.2.2 refers to a '13 million Nepali text corpus' and Table 3 lists the corpus size as 9.6 GB. The paper also says the GPT-2 architecture was initialized with 'zero weight initialisation,' which is not standard and is not further explained, and no model dimensions (number of layers, hidden size, attention heads, parameter count) are given. These inconsistencies prevent the reader from reproducing or even understanding what was trained.","section":"Section 3.1.2, Table 1; Section 3.2.2; Table 3"},{"comment":"The novelty claim that 'there isn't any generative language model for the Nepali language' is asserted without a systematic literature search. The literature review cites only NepaliBERT and NepBERTa, both BERT-style understanding models, but does not discuss generative models, fine-tuned multilingual LLMs, or concurrent Nepali GPT variants. If such a model exists, the paper's framing as the 'first' generative Nepali LM would need to be withdrawn, though the corpus and QA benchmark could still stand as contributions. The authors should either provide a more thorough survey or soften the claim.","section":"Section 1; Section 2"}],"minor_comments":[{"comment":"The metric name is consistently misspelled as 'ROGUE' in Section 4.3.1 and Table 8; the correct term is ROUGE.","section":"Throughout"},{"comment":"The headings 'Casual Coherence' and 'Casual Consistency' should be 'Causal Coherence' and 'Causal Consistency.'","section":"Section 4.3.2"},{"comment":"The filtering criterion 'checking if any alphanumeric character existed' is not a meaningful cleaning step, since nearly all Nepali text contains alphanumeric characters; the intended preprocessing should be described precisely.","section":"Section 3.1.1"},{"comment":"The paper mentions inference time recorded from HuggingFace API endpoints but gives no model identifier or repository link; adding a public model card would substantially improve reproducibility.","section":"Section 5.3"},{"comment":"The column 'Benchmark for English Dataset' is unexplained, and no source is given for the GPT-2 ROUGE values; the comparison in Section 6 to GPT-2's ROUGE-1 of 0.148 therefore lacks a verifiable reference.","section":"Table 8"},{"comment":"There is an unfinished phrase, 'othe utput of the Two of the experiments,' which should be corrected to 'the output of two of the experiments.'","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"I am not recommending reject because the training pipeline is plausible and the identified problems are, in principle, fixable in revision: the ROUGE contradiction can be resolved by reporting one correct value with a precise protocol, the human evaluation can be documented with sample sizes and agreement measures, and the corpus-size inconsistencies can be corrected. However, if the authors cannot reconcile the two ROUGE-1 values, or if the evaluation cannot be strengthened with baselines and variance estimates, the central empirical claim will remain unsupported and the manuscript may not be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a straightforward engineering report—train a GPT-2-style CLM on a scraped Nepali corpus, add a QA fine-tune dataset, report perplexity and a few human-judged scores. The artifact itself is genuinely new and useful for a language with thin NLP resources: a 9.3 GB Devanagari corpus, a 4,296-pair QA dataset, and a trained Nepali tokenizer are things people in the low-resource community would actually use. The paper's claim to be the first generative Nepali LM may well be right, but it's asserted rather than surveyed; the literature review lists only two BERT-style encoders and never shows a systematic search. That's a real weakness but a fixable one.\n\nThe bigger problem is internal: the abstract and Table 8 report ROUGE-1 as 0.2604, while the same table and the conclusion report 0.1501. Both are labeled ROUGE-1 for the same 50-sample evaluation, with no caveat about a different setting. Whatever the truth, the paper as written does not establish the headline performance claim. The evaluation is also thin in other ways: no variance or significance, a nonstandard half-sentence ROUGE protocol for the CLM, three annotators with no agreement measure, and no released model or code. None of these are fatal on their own—this is a resource paper, not a theoretical contribution—but the metric contradiction is load-bearing because it undermines the central quantitative claim. The self-citation pattern is fine: citing your own earlier NepaliBERT is appropriate here, though the novelty argument would be stronger with an actual search for concurrent generative Nepali models.\n\nWhere the paper stands after that: the data-collection and preprocessing details are concrete and reproducible in principle, the training curves look plausible, and the perplexity (26.32) is in a reasonable range for a small GPT-2 on a low-resource language. But without a public model, a corrected metric table, and at least one independent baseline, I would not rely on the reported numbers. For a serious referee, the paper belongs on a borderline: the artifact deserves attention, but the manuscript needs a revision that fixes the contradiction and releases the model. I'd rather see this resubmitted with those fixes than accepted as-is.\n\nFor peer review: yes, send it to referees—the resource value is real and the community needs to know what's available. For my own work, I'd cite it for the corpus and dataset, not for the model's claimed quality.","headline":"A useful artifact in an under-resourced language, but the reported metrics are internally inconsistent and the 'first generative Nepali LM' claim is asserted, not demonstrated; worth a referee's time only if the authors can fix the eval and release the model.","tokens_in":6825,"tokens_out":651,"would_cite":true,"duration_ms":8471,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes NepaliGPT, a from-scratch GPT-2-style causal language model for Nepali, reporting 26.32 perplexity and adding a 9.3 GB Devanagari corpus and a 4,296-pair QA benchmark.","keywords":["Nepali language","generative language model","causal language modeling","Devanagari script","byte-pair encoding","perplexity","question answering","low-resource NLP"],"falsifier":"Compile a release-dated list of generative language models that can produce Nepali text and check whether any was publicly available before 19 June 2025; the existence of even one such model falsifies the paper's novelty claim. A separate check: recompute ROUGE-1 on the released test set, since the paper reports 0.2604 in the abstract and 0.1501 in the conclusion, two values that cannot both be the same metric.","tokens_in":5904,"feed_emoji":"🇳🇵","tokens_out":8680,"duration_ms":78671,"temperature":0.7,"pith_summary":"This paper claims that no generative language model exists for Nepali, and that the gap can be filled with a comparatively small, from-scratch causal model. The authors build NepaliGPT by training a GPT-2-style transformer on the Devanagari Corpus, about 383 million tokens of scraped and translated Nepali text, then fine-tuning it on 4,296 Nepali question-answer pairs. They report 26.32 perplexity, ROUGE-1 around 0.26, and human ratings of roughly 81% causal coherence and 85% causal consistency, which they read as evidence that a native Nepali generative model is viable. A sympathetic reader would care because the claim, if true, gives Nepali speakers a generative model that produces Devanagari text directly rather than routing through English.","feed_headline":"NepaliGPT: first Nepali GPT-style model hits 26.3 perplexity","feed_subtitle":"A 9.3 GB Devanagari corpus and a 4,296-pair QA benchmark come with the model.","key_machinery":"The carrying mechanism is a causal language model: a transformer decoder that learns to predict the next token given the previous ones, trained from a randomly initialised GPT-2-style checkpoint on a custom byte-pair-encoding tokeniser over Devanagari script. The tokeniser, built to a vocabulary of about 10,000 subwords, keeps the vocabulary manageable for the script's large alphabet; pretraining on the 9.3 GB corpus supplies language knowledge; full fine-tuning on question-answer pairs turns that knowledge into a usable Nepali question-answering system.","core_discovery":"The paper's central claim is that a generative language model for Nepali is both missing and buildable, and that NepaliGPT is that model. On its own terms, the discovery is that a GPT-2-style causal language model trained from scratch on roughly 383 million Nepali tokens reaches a perplexity of 26.32, close to the 24.35 the authors cite for GPT-2 on English Wikipedia, while its fine-tuned question-answering variant receives human ratings of 81.25% causal coherence and 85.41% causal consistency. The same research introduces the Devanagari Corpus (about 9.3 GB) and a Nepali question-answer benchmark of 4,296 pairs, which the authors present as reusable resources so that later work does not have to start from raw scraped text.","pith_inferences":["If the same recipe transfers, other Devanagari-script or low-resource languages could get a GPT-style model from a few gigabytes of scraped text, a byte-pair tokeniser, and a modest GPU.","The paper's claim that no generative Nepali model exists is not established by its two-model literature review; a systematic sweep of model repositories would be the natural check, and its absence leaves the novelty claim hostage to timing.","The reported performance suggests that language-specific models do not necessarily need massive scale, which undercuts the assumption that low-resource languages must wait for large multilingual models.","A direct head-to-head against a fine-tuned multilingual model on the same Nepali QA benchmark would make the native-model advantage testable; the paper does not run that comparison."],"forward_implications":["If the results hold, Nepali can support native generative applications—chatbots, article drafting, and question answering—without an English translation step.","The Devanagari Corpus gives future work a ready-made pretraining resource, so later Nepali models do not have to scrape and clean text from scratch.","The 4,296-pair question-answer benchmark gives the field a fixed test set for comparing Nepali generative models.","Fine-tuning NepaliGPT on instruction-style data would plausibly produce an instruction-following Nepali assistant, as the paper itself suggests.","Bigger, more diverse corpora should lower perplexity further, since the larger of the two pretraining experiments already improves perplexity from 87 to 26."],"supporting_citations":[{"why":"Supplies the Transformer attention architecture that the NepaliGPT causal language model is built on.","marker":"[5]"},{"why":"One of the two prior Nepali BERT-style models cited as evidence that only understanding-oriented models existed before NepaliGPT.","marker":"[6]"},{"why":"The other prior Nepali BERT-style model; together with [6] it frames the claimed gap that NepaliGPT fills.","marker":"[7]"},{"why":"A large-scale Nepali text corpus the authors combine into the Devanagari Corpus used for pretraining.","marker":"[9]"},{"why":"A multilingual crawled corpus whose Nepali subset supplies over 100 million words for pretraining.","marker":"[10]"},{"why":"The library used to train the custom byte-pair-encoding tokeniser for Devanagari text.","marker":"[11]"},{"why":"Defines the byte-pair encoding algorithm that gives the tokeniser its vocabulary.","marker":"[12]"},{"why":"Provides the architecture and the English Wikipedia perplexity and ROUGE benchmarks used as comparison points.","marker":"[13]"}],"fun_headline_variants":["NepaliGPT: first Nepali GPT, 26.3 perplexity","No Nepali LLM before: NepaliGPT reaches 26.3 perplexity","Devanagari corpus and 4,296 QA pairs debut with NepaliGPT","First Nepali generative LM: 26.3 perplexity, ROUGE-1 0.26"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that no other generative language model for Nepali exists; the paper supports this only by mentioning two BERT-style models, so if any Nepali-capable generative model predates this work, the novelty claim collapses.","fun_headline_variants_meta":{"raw":{"variants":["NepaliGPT: first Nepali GPT, 26.3 perplexity","No Nepali LLM before: NepaliGPT reaches 26.3 perplexity","Devanagari corpus and 4,296 QA pairs debut with NepaliGPT","First Nepali generative LM: 26.3 perplexity, ROUGE-1 0.26"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000964,"raw_usage":{"total_tokens":4079,"prompt_tokens":899,"completion_tokens":3180,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":3085}},"tokens_in":515,"tokens_out":3180,"duration_ms":24392,"temperature":1.0,"reasoning_tokens":3085,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:26:35.186987+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compile a release-dated list of generative language models that can produce Nepali text and check whether any was publicly available before 19 June 2025; the existence of even one such model falsifies the paper's novelty claim. A separate check: recompute ROUGE-1 on the released test set, since the paper reports 0.2604 in the abstract and 0.1501 in the conclusion, two values that cannot both be the same metric.","supporting_citations":[{"cited_title":"Nepalibert: Pre-training of masked language model in nepali corpus","cited_arxiv_id":null,"evidence_quote":"One of the two prior Nepali BERT-style models cited as evidence that only understanding-oriented models existed before NepaliGPT."},{"cited_title":"Nepberta: Nepali language model trained in a large corpus","cited_arxiv_id":null,"evidence_quote":"The other prior Nepali BERT-style model; together with [6] it frames the claimed gap that NepaliGPT fills."},{"cited_title":"A large scale nepali text corpus","cited_arxiv_id":null,"evidence_quote":"A large-scale Nepali text corpus the authors combine into the Devanagari Corpus used for pretraining."},{"cited_title":"Language models are unsupervised multitask learners","cited_arxiv_id":null,"evidence_quote":"Provides the architecture and the English Wikipedia perplexity and ROUGE benchmarks used as comparison points."}],"review_version":2}