{"id":"58e2f971-c584-4f2c-a1c7-5c727f2b4384","arxiv_id":"2608.07727","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Dedicated monolingual models and tokenizers for Tamil, Telugu, Kannada, and Malayalam outperform a shared multilingual model and mGPT on tokenizer efficiency and most fine-tuned tasks, but the evaluation is single-run and partly unequal-budget.","lead":"This paper trains five small GPT-2-style language models for four Dravidian languages: one model per language and one model shared across all four. It compares their tokenizers and fine-tuned performance, finding that dedicated per-language models beat a large multilingual baseline on sentiment and name recognition, though the measurements are noisy.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The downstream claim that monolingual models outperform mGPT rests on a single fine-tuning run that the paper itself reports as unstable; seed-averaged evaluation is required.","rationale":"The reader's weakest assumption identifies exactly the vulnerability I find most load-bearing: the downstream superiority claim is built on a single fine-tuning run that the paper itself reports as unstable. My read of the full text confirms this. Section 4.3 reports one run per model and task; Section 6 explicitly states that a rerun at the same nominal seed produced a materially different ranking for Tamil and Kannada. The 24-example sentiment test set and the 100-example Kannada NER training set make the ranking highly sensitive to seed-dependent variation, so the abstract's claim that monolingual models outperform mGPT is not yet supported. I do not see a separate, more fundamental concern that would change the verdict: the tokenizer-efficiency measurements are deterministic and externally comparable, the architecture and data are described in sufficient detail, and the paper is unusually transparent about its limitations. The unequal pretraining budget for the multilingual model (Section 4.4) additionally weakens the mono-vs-multi BPB comparison, but that is not the central abstract claim. Given the acknowledged instability, CONDITIONAL remains the right verdict, with the condition being seed-averaged and larger-scale downstream evaluation. Therefore I recommend no change to the reader's verdict.","tokens_in":6231,"tokens_out":3095,"duration_ms":31497,"concrete_test":"Re-run the fine-tuning experiments of Table 4 for all four languages, comparing each monolingual model, the multilingual model, and mGPT with at least 5 seeds per model-task combination, using the same data splits and hyperparameters. Report the mean and 95% bootstrap confidence interval of the mono-minus-mGPT difference in IndicSentiment accuracy and WikiANN F1. If for any language the interval includes zero, the abstract's universal outperformance claim must be weakened. For additional robustness, re-evaluate on a larger labeled sentiment test set than the current 24 examples.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim that monolingual models outperform mGPT on sentiment classification and NER is supported only by Table 4, which reports one fine-tuning run per model and task. Section 4.3 and Section 6 acknowledge that a separate run at the same nominal seed produced a materially different ranking for Tamil and Kannada. This is a direct admission that the reported mono-vs-mGPT ordering is not reproducible under the paper's own conditions, likely due to nondeterminism in bf16/tf32 mixed-precision training combined with extremely small evaluation sets: IndicSentiment uses a self-partitioned 24-example test set, and WikiANN Kannada has only 100 training examples. With 24 test examples, a difference of 0.792 vs 0.458 corresponds to roughly 8 examples, so a small change in which examples are correctly classified can flip the ranking. The paper's own limitation statement says these numbers 'should be read as indicative, not definitive,' but the abstract states the outperformance as a finding. The tokenizer-efficiency results are measured deterministically on a fixed sample and are not affected by this concern, but the downstream transfer claim is load-bearing for the paper's practical conclusion and is not established by the reported experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper trains five GPT-2-scale causal language models from scratch for four Dravidian languages (Tamil, Telugu, Kannada, Malayalam): four monolingual models with dedicated 32K-tokenizer and one multilingual model with a shared 64K-tokenizer. It evaluates them against mGPT on perplexity, bits-per-byte, tokenizer fertility/compression, and downstream sentiment classification (IndicSentiment) and NER (WikiANN). The tokenizer-efficiency measurements show the dedicated 32K tokenizers have lower fertility and higher compression than XLM-R, mBERT, and mGPT on all four languages. The downstream experiments report that the monolingual models outperform mGPT on most tasks/languages, but the paper itself flags that these are single-run results and that a rerun at the same nominal seed produced a materially different ranking for Tamil and Kannada. The multilingual model's pretraining diverged, and the released checkpoint corresponds to an epoch-1 save, making cross-model comparisons unequal-budget.","tokens_in":6466,"tokens_out":2124,"duration_ms":20577,"significance":"If the downstream claim were established, the paper would provide a useful controlled comparison of dedicated vs. shared subword tokenizers and monolingual vs. joint multilingual pretraining for morphologically rich, non-Latin-script languages at a modest parameter scale. The tokenizer-fertility results are measured cleanly and are a credible, reproducible contribution: they show that a 32K per-language vocabulary can beat far larger multilingual vocabularies in fertility and compression, and they document a striking mGPT Kannada outlier (16.39 tokens/word). The intrinsic-quality results are suggestive but currently confounded by the unequal training budget of the multilingual model. However, the headline downstream claim is not yet supported because the fine-tuning evaluation is single-run and the paper admits instability on small test sets. With seed-averaging and a properly trained multilingual baseline, the paper's practical conclusions could be made firm, but that work is not present.","major_comments":[{"comment":"The abstract's claim that \"monolingual models outperform mGPT on sentiment classification and named entity recognition\" rests entirely on Table 4, which reports one fine-tuning run per model and task. Section 6 explicitly states that a separate run at the same nominal seed produced a materially different ranking for Tamil and Kannada. With only 24 held-out sentiment examples (IndicSentiment) and 100 training examples for Kannada NER, the reported gaps (e.g., Tamil sentiment 0.792 vs. 0.458) correspond to a handful of examples and can flip with minor run-to-run variation. Seed-averaged fine-tuning (at minimum 3–5 seeds) with significance or interval reporting is required before the mono-vs-mGPT ordering is established. As written, the downstream claim is not reproducible under the paper's own conditions.","section":"§4.3, Table 4, §6"},{"comment":"The multilingual model is not a fully trained baseline: its pretraining diverged, and the released checkpoint is the epoch-1 best-validation-loss save, roughly one third of the nominal training budget, while all monolingual models completed 3 epochs. This makes every mono-vs-multi comparison, including the BPB results in Table 2, an unequal-budget comparison. The paper acknowledges this asymmetry, but the conclusion still states that \"monolingual pretraining is more efficient than multilingual pretraining on every language\" based on BPB, which is misleading when the multilingual model has seen far less data. The analysis should either retrain the multilingual model to convergence or explicitly relabel all multi-vs-mono comparisons as preliminary and remove them from the conclusion's summary claims.","section":"§4.4, §6"},{"comment":"The Kannada NER evaluation uses only 100 training examples, and the reported F1 values (mono 0.268, mGPT 0.034) are near the floor for this task. With such a small training set, span-level F1 is extremely noisy and sensitive to the exact train split; the result does not provide strong evidence about model quality. The paper should either use a larger Kannada NER setting (e.g., augment with more labeled data or use cross-validation) or clearly state that the Kannada NER result is anecdotal and exclude it from the aggregated \"monolingual models outperform mGPT\" claim.","section":"§4.3, Table 4, WikiANN Kannada"}],"minor_comments":[{"comment":"The Kannada mGPT cell is rendered as \"3.9516.39\" with the two numbers concatenated; this appears to be a formatting error and should be \"3.95, 16.39\" or placed on separate lines.","section":"Table 3"},{"comment":"The abstract says \"it's not clear\" in a formal register; consider rephrasing to \"it remains unclear\" for consistency with the journal style.","section":"Abstract and Introduction"},{"comment":"The text states that \"the ranking is unambiguous\" for BPB, but the ranking is between two models where one is only epoch-1; please qualify this statement with the unequal-budget caveat already acknowledged in §4.4.","section":"§4.1"},{"comment":"The related work mentions \"Sarvam\" without a citation or reference; please add the appropriate citation or remove the sentence.","section":"§2"},{"comment":"The phrase \"benchmarked against an identically fine-tuned mGPT baseline\" is not fully precise because mGPT is a much larger model (1.7B parameters) and its fine-tuning recipe may not be truly identical in effective batch size or learning rate; specify the mGPT parameter count and any adjustments.","section":"§3.4"}],"recommendation":"major_revision","confidential_remarks":"The tokenizer-fertility contribution is solid and likely publishable as a controlled study. The downstream claim is the advertised headline, but it is not yet supported because of single-run fine-tuning, acknowledged instability, and the unequal-budget multilingual baseline. I would not reject outright since the issues are fixable with additional experiments (seed-averaging, retraining or clearly labeling the multilingual baseline), but the revision must either add those experiments or weaken the abstract's claim to match the evidence. The paper is at the boundary of scope for a serious journal; if the downstream claim cannot be substantiated, the tokenizer study alone may be more appropriate for a workshop or a specialized resource paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the tokenizer-efficiency half of this paper is real and worth borrowing; the downstream half is not established, and the paper tells you so itself.\n\nWhat's actually new: a matched set of from-scratch GPT-2-scale models for Tamil, Telugu, Kannada, and Malayalam, with dedicated 32K tokenizers versus a joint 64K tokenizer. The fertility and compression measurements against mBERT, XLM-R, and mGPT are clean and externally meaningful, and the mGPT Kannada outlier (16.4 tokens/word) is a concrete finding. The BPB analysis is the right way to correct for tokenizer differences, and the discussion of why raw PPL misleads here is sound. The paper is unusually candid: section 4.4 openly reports that the multilingual model diverged and that the released checkpoint is only epoch-1, and section 6 says the downstream numbers \"should be read as indicative, not definitive.\"\n\nSoft spots, in order of size. First, the abstract and conclusion state that monolingual models \"outperform mGPT\" as a finding, but that claim rests on Table 4: one fine-tuning run per model and task, a self-partitioned 24-example sentiment test set, a 100-example Kannada NER training split, and the paper's own admission that a same-seed rerun produced a materially different ranking for Tamil and Kannada. With 24 test examples, a few flips change the ordering. So the downstream headline is not reproducible on the paper's own evidence. Second, the mono-versus-multi comparison is unequal-budget: the multilingual model effectively got one epoch, so the BPB and fertility advantages might partly reflect training budget, not just tokenization. The author acknowledges this, but it limits the strength of the claim. Third, no code or checkpoints are released, which makes the instability hard for others to chase down.\n\nWho gets value: people working on low-resource tokenizers and Dravidian NLP will want the tokenizer table and the BPB analysis; the paper is also a useful cautionary example of how single-run fine-tuning plus tiny test sets can produce confident-sounding conclusions that do not stick.\n\nMy call: worth a serious referee, but the review should push for seed-averaged fine-tuning, a properly labeled sentiment test set, and an honestly trained multilingual baseline. The tokenizer-efficiency core can survive that; the downstream claim as written should not.","headline":"Clean tokenizer-efficiency study whose headline downstream claim is undercut by single-run fine-tuning and an unequal-budget multilingual baseline that the paper itself admits.","tokens_in":7013,"tokens_out":2611,"would_cite":false,"duration_ms":24323,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For Tamil, Telugu, Kannada, and Malayalam, dedicated monolingual GPT-2 models beat mGPT on sentiment and NER, and dedicated tokenizers are more efficient.","keywords":["Dravidian languages","monolingual language models","multilingual models","tokenizer efficiency","bits-per-byte","subword fertility","low-resource NLP","GPT-2"],"falsifier":"Rerun the Table 4 fine-tuning protocol with multiple seeds, say 10, and report seed-averaged accuracy and F1 with confidence intervals; if the monolingual advantage over mGPT vanishes or flips for Tamil or Kannada, the central downstream claim fails.","tokens_in":6006,"feed_emoji":"🧠","tokens_out":6873,"duration_ms":60912,"temperature":0.7,"pith_summary":"Low-resource Dravidian languages are underrepresented in multilingual training mixes, so their per-language ability in general-purpose models is uncertain. This paper tries to establish that for Tamil, Telugu, Kannada, and Malayalam, a from-scratch monolingual GPT-2-small model with a dedicated tokenizer outperforms the multilingual mGPT on sentiment classification and named entity recognition, and that its tokenizer is more efficient than shared multilingual vocabularies. The author also argues that bits-per-byte, not raw perplexity, is the fair comparison when tokenizers differ. If correct, it means cheap, language-specific pretraining can recover real quality that large multilingual models leave on the table for morphologically dense non-Latin-script languages.","feed_headline":"Per-language models beat mGPT for Dravidian languages","feed_subtitle":"Even at 110M parameters, monolingual GPT-2 models win on sentiment, NER, and tokenizer efficiency.","key_machinery":"The argument turns on two linked objects: a matched family of GPT-2-small decoder-only models (12 layers, 768 hidden, 110M parameters per language, 135M for the joint model) and SentencePiece BPE vocabularies, with a 32K vocabulary dedicated to each language and a shared 64K vocabulary for a joint multilingual model. The measurement that carries the comparison is bits-per-byte (BPB), which normalizes raw cross-entropy by tokenizer output length, plus subword fertility (tokens per word) and compression (bytes per token). This is what lets the author separate tokenizer quality from language-model quality and argue that lower raw perplexity for the multilingual model is an artifact of a tokenizer that produces more, cheaper tokens per byte.","core_discovery":"The author's central claim is that for Dravidian languages, dedicated monolingual GPT-2 models trained from scratch with per-language 32K subword tokenizers recover quality that massively multilingual models dilute: they beat mGPT on IndicSentiment sentiment classification and WikiANN named entity recognition in nearly every language tested, and their tokenizers are more efficient than XLM-R, mBERT, and mGPT tokenizers and than the paper's own 64K joint tokenizer on bits-per-byte. The paper interprets bits-per-byte as the tokenizer-agnostic metric, so monolingual pretraining is more efficient on all four languages even when raw perplexity favors the multilingual model on Telugu and Tamil. It also reports that mGPT's tokenizer has very poor Kannada coverage, at 16.39 tokens per word. The paper explicitly cautions that the downstream numbers are single-run and that a repeat run at the same nominal seed changed the Tamil and Kannada ranking; it also notes the multilingual model's released checkpoint is from an epoch-1 best-validation save after a mid-training divergence, making the comparison unequal-budget.","pith_inferences":["I would treat the downstream ranking as a hypothesis until seed-averaged runs are published, since the paper's own repeated run changed the Tamil and Kannada ordering.","If the BPB and fertility findings hold under sliding-window evaluation, dedicated tokenization may be the main source of the quality gain, separable from monolingual pretraining data.","The same matched design could be applied to other low-resource, script-distinct families, such as other Indic scripts, to see whether per-language pretraining beats multilingual dilution at equal parameter budget.","A direct next experiment is to hold the tokenizer fixed and vary the pretraining language mix, isolating whether monolingual data or dedicated vocabulary drives the observed advantage."],"forward_implications":["For agglutinative Brahmic-script languages, dedicated 32K BPE tokenizers sharply reduce tokens per word relative to mGPT, so a fixed 1024-token context window holds substantially more real text.","A model family at 110M–135M parameters, trained only on public Dravidian corpora, can outperform a much larger multilingual baseline on sentiment and NER when data and tokenizer are language-matched.","Bits-per-byte should replace raw perplexity when comparing models with different tokenizers; the paper's BPB ranking is monolingual-first on all four languages.","The joint 64K tokenizer, though trained specifically on this language family, still underperforms dedicated tokenizers, so sharing a vocabulary across related but script-distinct languages carries a measurable cost.","mGPT's near-zero effective Kannada-script coverage is a concrete example of per-language dilution in massively multilingual models."],"supporting_citations":[{"why":"mGPT is the multilingual baseline that the fine-tuned downstream comparisons must beat.","marker":"(Shliazhko et al., 2024)"},{"why":"Provides the CC-100 training corpus and XLM-R tokenizer used as a fertility baseline.","marker":"(Conneau et al., 2020)"},{"why":"Provides the Samanantar parallel corpora used as one of the pretraining sources.","marker":"(Ramesh et al., 2022)"},{"why":"Supplies the claim that multilingual tokenizers allocate efficiency unevenly across languages, motivating dedicated tokenizers.","marker":"(Petrov et al., 2023)"},{"why":"Shows multilingual tokenizers underperform monolingual ones, the effect the paper reproduces for Dravidian languages.","marker":"(Rust et al., 2021)"},{"why":"Provides IndicNLPSuite and IndicBERT, the Indic evaluation and tokenization resources the study builds on.","marker":"(Kakwani et al., 2020)"}],"fun_headline_variants":["Monolingual Dravidian GPT-2s beat mGPT on sentiment and NER","Tamil, Telugu, Kannada, Malayalam: dedicated models win","From-scratch Dravidian models top mGPT tokenizer efficiency","110M monolingual models outdo mGPT for Dravidian"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that monolingual models beat mGPT on downstream tasks rests on a single fine-tuning run per task and language; a second run at the same nominal seed shifted the Tamil and Kannada rankings, so the downstream advantage is not yet stable.","fun_headline_variants_meta":{"raw":{"variants":["Monolingual Dravidian GPT-2s beat mGPT on sentiment and NER","Tamil, Telugu, Kannada, Malayalam: dedicated models win","From-scratch Dravidian models top mGPT tokenizer efficiency","110M monolingual models outdo mGPT for Dravidian"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000553,"raw_usage":{"total_tokens":2640,"prompt_tokens":954,"completion_tokens":1686,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":570,"completion_tokens_details":{"reasoning_tokens":1602}},"tokens_in":570,"tokens_out":1686,"duration_ms":10924,"temperature":1.0,"reasoning_tokens":1602,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T00:21:30.541137+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the Table 4 fine-tuning protocol with multiple seeds, say 10, and report seed-averaged accuracy and F1 with confidence intervals; if the monolingual advantage over mGPT vanishes or flips for Tamil or Kannada, the central downstream claim fails.","supporting_citations":[],"review_version":1}