Pith. sign in

REVIEW 3 major objections 5 minor 22 references

Evaluating Dedicated Monolingual and Joint Multilingual Causal Models for Dravidian Languages

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For Tamil, Telugu, Kannada, and Malayalam, dedicated monolingual GPT-2 models beat mGPT on sentiment and NER, and dedicated tokenizers are more efficient.

desk verdict Clean tokenizer-efficiency study whose headline downstream claim is undercut by single-run fine-tuning and an unequal-budget multilingual baseline that the paper itself admits. read the letter →

arxiv 2608.07727 v1 pith:YSLS67TD submitted 2026-08-07 cs.CL

Venkata Naga Sai Vishnu Rohit Pulipaka This is my paper · ORCID
classification cs.CL
keywords Dravidianlanguagesmonolinguallanguagemodelsmultilingualtokenizerefficiencybits-per-bytesubwordfertilitylow-resourceNLPGPT-2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Low-resource Dravidian languages are underrepresented in multilingual training mixes, so their per-language ability in general-purpose models is uncertain. This paper tries to establish that for Tamil, Telugu, Kannada, and Malayalam, a from-scratch monolingual GPT-2-small model with a dedicated tokenizer outperforms the multilingual mGPT on sentiment classification and named entity recognition, and that its tokenizer is more efficient than shared multilingual vocabularies. The author also argues that bits-per-byte, not raw perplexity, is the fair comparison when tokenizers differ. If correct, it means cheap, language-specific pretraining can recover real quality that large multilingual models leave on the table for morphologically dense non-Latin-script languages.

What carries the argument

The argument turns on two linked objects: a matched family of GPT-2-small decoder-only models (12 layers, 768 hidden, 110M parameters per language, 135M for the joint model) and SentencePiece BPE vocabularies, with a 32K vocabulary dedicated to each language and a shared 64K vocabulary for a joint multilingual model. The measurement that carries the comparison is bits-per-byte (BPB), which normalizes raw cross-entropy by tokenizer output length, plus subword fertility (tokens per word) and compression (bytes per token). This is what lets the author separate tokenizer quality from language-model quality and argue that lower raw perplexity for the multilingual model is an artifact of a tokenizer that produces more, cheaper tokens per byte.

What would settle it

Rerun the Table 4 fine-tuning protocol with multiple seeds, say 10, and report seed-averaged accuracy and F1 with confidence intervals; if the monolingual advantage over mGPT vanishes or flips for Tamil or Kannada, the central downstream claim fails.

Watch

Extended reading notes

Core claim

The author's central claim is that for Dravidian languages, dedicated monolingual GPT-2 models trained from scratch with per-language 32K subword tokenizers recover quality that massively multilingual models dilute: they beat mGPT on IndicSentiment sentiment classification and WikiANN named entity recognition in nearly every language tested, and their tokenizers are more efficient than XLM-R, mBERT, and mGPT tokenizers and than the paper's own 64K joint tokenizer on bits-per-byte. The paper interprets bits-per-byte as the tokenizer-agnostic metric, so monolingual pretraining is more efficient on all four languages even when raw perplexity favors the multilingual model on Telugu and Tamil. It also reports that mGPT's tokenizer has very poor Kannada coverage, at 16.39 tokens per word. The paper explicitly cautions that the downstream numbers are single-run and that a repeat run at the same nominal seed changed the Tamil and Kannada ranking; it also notes the multilingual model's released checkpoint is from an epoch-1 best-validation save after a mid-training divergence, making the comparison unequal-budget.

Load-bearing premise

The claim that monolingual models beat mGPT on downstream tasks rests on a single fine-tuning run per task and language; a second run at the same nominal seed shifted the Tamil and Kannada rankings, so the downstream advantage is not yet stable.

Editorial extensions

If this is right

  • For agglutinative Brahmic-script languages, dedicated 32K BPE tokenizers sharply reduce tokens per word relative to mGPT, so a fixed 1024-token context window holds substantially more real text.
  • A model family at 110M–135M parameters, trained only on public Dravidian corpora, can outperform a much larger multilingual baseline on sentiment and NER when data and tokenizer are language-matched.
  • Bits-per-byte should replace raw perplexity when comparing models with different tokenizers; the paper's BPB ranking is monolingual-first on all four languages.
  • The joint 64K tokenizer, though trained specifically on this language family, still underperforms dedicated tokenizers, so sharing a vocabulary across related but script-distinct languages carries a measurable cost.
  • mGPT's near-zero effective Kannada-script coverage is a concrete example of per-language dilution in massively multilingual models.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • I would treat the downstream ranking as a hypothesis until seed-averaged runs are published, since the paper's own repeated run changed the Tamil and Kannada ordering.
  • If the BPB and fertility findings hold under sliding-window evaluation, dedicated tokenization may be the main source of the quality gain, separable from monolingual pretraining data.
  • The same matched design could be applied to other low-resource, script-distinct families, such as other Indic scripts, to see whether per-language pretraining beats multilingual dilution at equal parameter budget.
  • A direct next experiment is to hold the tokenizer fixed and vary the pretraining language mix, isolating whether monolingual data or dedicated vocabulary drives the observed advantage.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper trains five GPT-2-scale causal language models from scratch for four Dravidian languages (Tamil, Telugu, Kannada, Malayalam): four monolingual models with dedicated 32K-tokenizer and one multilingual model with a shared 64K-tokenizer. It evaluates them against mGPT on perplexity, bits-per-byte, tokenizer fertility/compression, and downstream sentiment classification (IndicSentiment) and NER (WikiANN). The tokenizer-efficiency measurements show the dedicated 32K tokenizers have lower fertility and higher compression than XLM-R, mBERT, and mGPT on all four languages. The downstream experiments report that the monolingual models outperform mGPT on most tasks/languages, but the paper itself flags that these are single-run results and that a rerun at the same nominal seed produced a materially different ranking for Tamil and Kannada. The multilingual model's pretraining diverged, and the released checkpoint corresponds to an epoch-1 save, making cross-model comparisons unequal-budget.

Significance. If the downstream claim were established, the paper would provide a useful controlled comparison of dedicated vs. shared subword tokenizers and monolingual vs. joint multilingual pretraining for morphologically rich, non-Latin-script languages at a modest parameter scale. The tokenizer-fertility results are measured cleanly and are a credible, reproducible contribution: they show that a 32K per-language vocabulary can beat far larger multilingual vocabularies in fertility and compression, and they document a striking mGPT Kannada outlier (16.39 tokens/word). The intrinsic-quality results are suggestive but currently confounded by the unequal training budget of the multilingual model. However, the headline downstream claim is not yet supported because the fine-tuning evaluation is single-run and the paper admits instability on small test sets. With seed-averaging and a properly trained multilingual baseline, the paper's practical conclusions could be made firm, but that work is not present.

major comments (3)
  1. [§4.3, Table 4, §6] The abstract's claim that "monolingual models outperform mGPT on sentiment classification and named entity recognition" rests entirely on Table 4, which reports one fine-tuning run per model and task. Section 6 explicitly states that a separate run at the same nominal seed produced a materially different ranking for Tamil and Kannada. With only 24 held-out sentiment examples (IndicSentiment) and 100 training examples for Kannada NER, the reported gaps (e.g., Tamil sentiment 0.792 vs. 0.458) correspond to a handful of examples and can flip with minor run-to-run variation. Seed-averaged fine-tuning (at minimum 3–5 seeds) with significance or interval reporting is required before the mono-vs-mGPT ordering is established. As written, the downstream claim is not reproducible under the paper's own conditions.
  2. [§4.4, §6] The multilingual model is not a fully trained baseline: its pretraining diverged, and the released checkpoint is the epoch-1 best-validation-loss save, roughly one third of the nominal training budget, while all monolingual models completed 3 epochs. This makes every mono-vs-multi comparison, including the BPB results in Table 2, an unequal-budget comparison. The paper acknowledges this asymmetry, but the conclusion still states that "monolingual pretraining is more efficient than multilingual pretraining on every language" based on BPB, which is misleading when the multilingual model has seen far less data. The analysis should either retrain the multilingual model to convergence or explicitly relabel all multi-vs-mono comparisons as preliminary and remove them from the conclusion's summary claims.
  3. [§4.3, Table 4, WikiANN Kannada] The Kannada NER evaluation uses only 100 training examples, and the reported F1 values (mono 0.268, mGPT 0.034) are near the floor for this task. With such a small training set, span-level F1 is extremely noisy and sensitive to the exact train split; the result does not provide strong evidence about model quality. The paper should either use a larger Kannada NER setting (e.g., augment with more labeled data or use cross-validation) or clearly state that the Kannada NER result is anecdotal and exclude it from the aggregated "monolingual models outperform mGPT" claim.
minor comments (5)
  1. [Table 3] The Kannada mGPT cell is rendered as "3.9516.39" with the two numbers concatenated; this appears to be a formatting error and should be "3.95, 16.39" or placed on separate lines.
  2. [Abstract and Introduction] The abstract says "it's not clear" in a formal register; consider rephrasing to "it remains unclear" for consistency with the journal style.
  3. [§4.1] The text states that "the ranking is unambiguous" for BPB, but the ranking is between two models where one is only epoch-1; please qualify this statement with the unequal-budget caveat already acknowledged in §4.4.
  4. [§2] The related work mentions "Sarvam" without a citation or reference; please add the appropriate citation or remove the sentence.
  5. [§3.4] The phrase "benchmarked against an identically fine-tuned mGPT baseline" is not fully precise because mGPT is a much larger model (1.7B parameters) and its fine-tuning recipe may not be truly identical in effective batch size or learning rate; specify the mGPT parameter count and any adjustments.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper is an empirical evaluation whose claims rest on external benchmarks and held-out splits, not on a derivation that reduces to its inputs.

full rationale

This paper contains no derivation chain, no fitted parameters presented as predictions, no self-citations, and no imported uniqueness or ansatz results. The central comparisons are empirical: the author trains dedicated monolingual and a shared multilingual GPT-2-style model and evaluates them on held-out perplexity/BPB, measured tokenizer fertility against external tokenizers, and downstream fine-tuning on IndicSentiment and WikiANN with an mGPT baseline. Each of these evaluations uses data or benchmarks external to the training objective, so the results are not equivalent to the inputs by construction. The tokenizer-efficiency result compares trained SentencePiece tokenizers against mBERT, XLM-R, and mGPT tokenizers on a fixed 2,000-sentence sample; while it is expected that a tokenizer trained on a language is more fertile for that language, the comparison is an empirical measurement, not a definitional identity. The paper's own limitation statement acknowledges that the downstream fine-tuning numbers are single-run and that a separate run at the same nominal seed produced a materially different ranking for Tamil and Kannada; this is a validity and reproducibility concern, not a circularity concern. Similarly, the self-partitioned 24-example sentiment test set weakens statistical confidence but does not make the evaluation circular. No circular steps are present.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The ledger contains no invented entities. The central empirical comparisons rely on domain assumptions about data representativeness, baseline validity, and single-run stability. The multilingual model's early stopping is the main ad hoc asymmetry, disclosed by the author, not a hidden fitting step.

free parameters (3)
  • Monolingual tokenizer vocabulary size (32K) = 32,000
    Chosen by hand for each monolingual model; the tokenizer efficiency comparison depends on this size versus the 64K joint and larger multilingual vocabularies.
  • Joint multilingual tokenizer vocabulary size (64K) = 64,000
    Chosen by hand for the shared model; the dedicated-vs-shared tokenizer claim is a comparison between different chosen vocabulary sizes.
  • Pretraining and fine-tuning hyperparameters = Pretrain: batch 256, LR 1e-4, warmup 4000, 3 epochs; downstream: 5 epochs, LR 2e-5, batch 16
    Fixed by hand, not tuned against the evaluation metrics, but they define the training budget. The unequal budget caused by the multilingual model's early stopping is the main consequence.
assumptions (5)
  • domain assumption CC-100, Wikipedia, and Samanantar target-side text are clean and representative enough for each language.
    Section 3.1 uses these sources with only line-level filtering and no deduplication; noisy or misaligned data would shift intrinsic quality scores.
  • domain assumption mGPT is a meaningful multilingual baseline for fine-tuning comparisons.
    Sections 2 and 3.4 pair dedicated 110M-135M models against mGPT despite likely differences in scale and training data; mGPT's 16.39 tokens/word fertility for Kannada makes that baseline especially weak.
  • domain assumption One fine-tuning run per model and task estimates model ranking.
    Section 3.4 and Table 4 use single runs; section 6 reports a same-seed rerun changed Tamil and Kannada rankings.
  • domain assumption PPL and BPB on 1024-token truncations faithfully measure language quality.
    Section 3.4 truncates each test line to 1024 tokens without sliding windows, which can bias perplexity for longer documents.
  • ad hoc to paper The epoch-1 multilingual checkpoint represents the multilingual model in comparisons.
    Section 4.4 discloses the training divergence and that the released model is an epoch-1 save, about one third of the planned budget; mono-vs-multi comparisons inherit this asymmetry.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating Dedicated Monolingual and Joint Multilingual Causal Models for Dravidian Languages." pith.science (2026). https://pith.science/paper/YSLS67TD

@misc{pith2026260807727,
  author       = {Pith},
  title        = {Pith review of: Evaluating Dedicated Monolingual and Joint Multilingual Causal Models for Dravidian Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YSLS67TD}},
  note         = {Machine review of arXiv:2608.07727}
}
read the original abstract

Dravidian languages, mainly Tamil, Telugu, Kannada, and Malayalam make up only a small part of the data used to train multilingual language models, so it's not clear how much per-language ability these models actually keep. I have trained five GPT-2 architecture models from scratch to compare four monolingual models (one each for Tamil, Telugu, Kannada, and Malayalam, each with its own 32K-vocabulary subword tokenizer) against one multilingual model sharing a 64K-vocabulary subword tokenizer across all four languages. All the 5 models are trained on cleaned CC-100, Wikipedia, and Samanantar data. I have tested the models on perplexity, bits-per-byte, tokenizer efficiency, and fine-tuning results which are compared against mGPT. The monolingual models outperform mGPT on sentiment classification and named entity recognition, and their tokenizers proved more efficient than the shared multilingual model across all the languages tested.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 7 canonical work pages

  1. [2]

    Wissam Antoun, Fady Baly, and Hazem Hajj. 2020. https://aclanthology.org/2020.osact-1.2/ A ra BERT : Transformer-based model for A rabic language understanding . In Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection, pages 9--15, Marseille, France. European Language Resour...

    work page 2020

  2. [7]

    Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha Talukdar. 2021. https://arxiv.org/abs/2103.10730 Muril: Multilingual representations for indian languages . Preprint, arXiv:2103.10730

    arXiv 2021

  3. [9]

    Aleksandar Petrov, Emanuele La Malfa, Philip Torr, and Adel Bibi. 2023. https://openreview.net/forum?id=78yDLKi95p Language model tokenizers introduce unfairness between languages . In Thirty-seventh Conference on Neural Information Processing Systems

    2023

  4. [10]

    Gowtham Ramesh, Sumanth Doddapaneni, Aravinth Bheemaraj, Mayank Jobanputra, Raghavan AK, Ajitesh Sharma, Sujit Sahoo, Harshita Diddee, Mahalakshmi J, Divyanshu Kakwani, Navneet Kumar, Aswin Pradeep, Srihari Nagaraj, Kumar Deepak, Vivek Raghavan, Anoop Kunchukuttan, Pratyush Kumar, and Mitesh Shantadevi Khapra. 2022. https://doi.org/10.1162/tacl_a_00452 Sa...

    doi:10.1162/tacl_a_00452 2022

  5. [13]

    Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. 2019. https://arxiv.org/abs/1912.07076 Multilingual is not enough: Bert for finnish . Preprint, arXiv:1912.07076

    arXiv 2019

  6. [14]

    BigScience Workshop. 2024. http://jmlr.org/papers/v25/23-0581.html Bloom: A 176b-parameter open-access multilingual language model . Journal of Machine Learning Research, 25(422):1--74

    work page 2024

  7. [16]

    BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding

    Devlin, Jacob and Chang, Ming-Wei and Lee, Kenton and Toutanova, Kristina. BERT : Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers). 2019. doi:10.18653/v...

    doi:10.18653/v1/n19-1423 2019

  8. [17]

    Unsupervised Cross-lingual Representation Learning at Scale

    Conneau, Alexis and Khandelwal, Kartikay and Goyal, Naman and Chaudhary, Vishrav and Wenzek, Guillaume and Guzm \'a n, Francisco and Grave, Edouard and Ott, Myle and Zettlemoyer, Luke and Stoyanov, Veselin. Unsupervised Cross-lingual Representation Learning at Scale. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. ...

    doi:10.18653/v1/2020.acl-main.747 2020

Show all 22 references
  1. [18]

    m T 5: A Massively Multilingual Pre-trained Text-to-Text Transformer

    Xue, Linting and Constant, Noah and Roberts, Adam and Kale, Mihir and Al-Rfou, Rami and Siddhant, Aditya and Barua, Aditya and Raffel, Colin. m T 5: A Massively Multilingual Pre-trained Text-to-Text Transformer. Proceedings of the 2021 Conference of the North American Chapter ...

    2021 doi

  2. [19]

    m GPT : Few-Shot Learners Go Multilingual

    Shliazhko, Oleh and Fenogenova, Alena and Tikhonova, Maria and Kozlova, Anastasia and Mikhailov, Vladislav and Shavrina, Tatiana. m GPT : Few-Shot Learners Go Multilingual. Transactions of the Association for Computational Linguistics. 2024. doi:10.1162/tacl_a_00633

    2024 doi

  3. [20]

    and Kumar, Pratyush

    Kakwani, Divyanshu and Kunchukuttan, Anoop and Golla, Satish and N.C., Gokul and Bhattacharyya, Avik and Khapra, Mitesh M. and Kumar, Pratyush. I ndic NLPS uite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for I ndian Languages. Find...

    2020 doi

  4. [21]

    arXiv preprint arXiv:2305.16307 , year=

    Indictrans2: Towards high-quality and accessible machine translation models for all 22 scheduled indian languages , author=. arXiv preprint arXiv:2305.16307 , year=

    arXiv

  5. [22]

    Thirty-seventh Conference on Neural Information Processing Systems , year=

    Language Model Tokenizers Introduce Unfairness Between Languages , author=. Thirty-seventh Conference on Neural Information Processing Systems , year=

  6. [23]

    How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models

    Rust, Phillip and Pfeiffer, Jonas and Vuli \'c , Ivan and Ruder, Sebastian and Gurevych, Iryna. How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Models. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics a...

    2021 doi

  7. [24]

    I ndic XNLI : Evaluating Multilingual Inference for I ndian Languages

    Aggarwal, Divyanshu and Gupta, Vivek and Kunchukuttan, Anoop. I ndic XNLI : Evaluating Multilingual Inference for I ndian Languages. Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing. 2022. doi:10.18653/v1/2022.emnlp-main.755

    2022 doi

  8. [25]

    2021 , eprint=

    MuRIL: Multilingual Representations for Indian Languages , author=. 2021 , eprint=

    2021

  9. [26]

    C amem BERT : a Tasty F rench Language Model

    Martin, Louis and Muller, Benjamin and Ortiz Su \'a rez, Pedro Javier and Dupont, Yoann and Romary, Laurent and de la Clergerie, \'E ric and Seddah, Djam \'e and Sagot, Beno \^i t. C amem BERT : a Tasty F rench Language Model. Proceedings of the 58th Annual Meeting of the Asso...

    2020 doi

  10. [27]

    2019 , eprint=

    Multilingual is not enough: BERT for Finnish , author=. 2019 , eprint=

    2019

  11. [28]

    A ra BERT : Transformer-based Model for A rabic Language Understanding

    Antoun, Wissam and Baly, Fady and Hajj, Hazem. A ra BERT : Transformer-based Model for A rabic Language Understanding. Proceedings of the 4th Workshop on Open-Source Arabic Corpora and Processing Tools, with a Shared Task on Offensive Language Detection. 2020

    2020

  12. [29]

    Wikimedia Downloads

    Wikimedia Foundation. Wikimedia Downloads

  13. [30]

    Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 I ndic Languages

    Ramesh, Gowtham and Doddapaneni, Sumanth and Bheemaraj, Aravinth and Jobanputra, Mayank and AK, Raghavan and Sharma, Ajitesh and Sahoo, Sujit and Diddee, Harshita and J, Mahalakshmi and Kakwani, Divyanshu and Kumar, Navneet and Pradeep, Aswin and Nagaraj, Srihari and Deepak, K...

    2022

  14. [31]

    Journal of Machine Learning Research , year =

    BigScience Workshop , title =. Journal of Machine Learning Research , year =

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.