Pith. sign in

REVIEW 4 major objections 5 minor 16 references

BERT-based Models vs. Large Language Models for Low-Resource Named Entity Recognition: A Comparative Study on Marathi

T0 review · 4 major / 5 minor · reviewed 2026-07-31 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuned MahaBERT variants beat zero-shot LLMs by more than 20 F1 points on Marathi named entity recognition, and matching training augmentation to test structure matters.

desk verdict Fine-tuned MahaBERT plausibly beats zero-shot LLMs on Marathi NER by ~0.20 F1, but the paper's unspecified token alignment and missing variance estimates need fixes before the exact numbers are trustworthy. read the letter →

arxiv 2607.23344 v1 pith:VQNB6G3C submitted 2026-07-25 cs.CL cs.LG

Hariom Ingle , Ronit Ghode , Ishwari Gondkar , Jidnyasa Harad , Raviraj Joshi This is my paper · ORCID
classification cs.CLcs.LG
keywords MarathiNERnamedentityrecognitionMahaBERTfine-tuninglargelanguagemodelszero-shotevaluationlow-resourcelanguagesdataaugmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether large, general-purpose language models have made specialized fine-tuned models obsolete for named-entity recognition in a low-resource language like Marathi. It fine-tunes MahaBERT on the MahaNER corpus, with and without sentence-repetition and random-concatenation augmentation, and compares those models against Gemini, LLaMA-3.3-70B, and two Gemma variants on the same 500 Marathi test sentences. The reported results are decisive: the best MahaBERT model reaches F1 0.91 while the best LLM, Gemini, reaches only 0.69–0.70, a gap of more than 20 points. The paper also finds that training with a given augmentation strategy improves performance on test sets built the same way, and that LLMs tend toward high recall but low precision, over-predicting named entities. The authors read this as evidence that language-specific, task-specific supervision still beats general-purpose zero-shot inference for low-resource NER.

What carries the argument

MahaBERT, a Marathi-specific BERT encoder fine-tuned with a token-classification head over non-IOB labels, is the central object. The augmentation strategies — self-concatenation (repeating the same sentence 4x or 10x) and random concatenation (combining different sentences) — are the mechanism used to study context-length effects. For LLMs the key procedural device is token alignment, which maps free-form LLM output back to gold token labels; the comparison's fairness depends on this step.

What would settle it

Run the same 500-sentence test set through Gemini without any output alignment and compare the raw output sequence to gold labels using strict token matching; if unaligned F1 is much below the reported 0.69, the alignment heuristic is what carries the LLM score. A second check: give the LLM 10–20 labelled Marathi examples in the prompt and see whether the F1 gap shrinks.

Watch

Extended reading notes

Core claim

The central claim is that fine-tuned MahaBERT models outperform both the MahaNER baseline (F1 0.8843) and all evaluated zero-shot LLMs on every one of the three test variants. On the standard test set the MahaNER base model scores F1 0.91; the normal-repeat-4x and random-repeat-4x variants reach 0.90–0.91 on their matching test sets. The best LLM, Gemini, scores 0.69–0.70 across test sets, with LLaMA-3.3-70B and the Gemma models lower. The authors attribute the gap to task-specific training on Marathi data and to LLMs' unreliable token-level output formatting, which hurts precision.

Load-bearing premise

The load-bearing premise is that the token-alignment procedure for LLM outputs is unbiased — that it does not systematically discard inaccurate tokens or repair malformed spans in a way that inflates LLM F1 scores; the paper says alignment was enforced (Section 5.2) but never describes the algorithm.

Editorial extensions

If this is right

  • For Marathi NER on news text, a general-purpose LLM used zero-shot should not be expected to replace a fine-tuned language-specific model; the gap is larger than what prompt tuning usually recovers.
  • Training on longer sequences via either repetition or random concatenation changes which test distribution a model handles best, suggesting test data structure should be matched to training augmentation.
  • LLMs' over-prediction of entities (high recall, low precision) means their raw outputs need precision-oriented post-processing before they are usable in production NER pipelines.
  • The success of MahaBERT over multilingual baselines indicates language-specific pretraining remains valuable for low-resource NER.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test the authors did not run is few-shot prompting: giving LLMs a handful of labelled Marathi examples might close part of the 20-point gap, since the paper only evaluates zero-shot.
  • The token-alignment step is under-specified; if alignment discarded unlabelled tokens or fuzzy-matched spans, the LLM scores could be biased. Reporting alignment details or error analysis per token would strengthen confidence in the comparison.
  • Because the test set is limited to 500 sentences from one news domain, the 'low-resource NER' conclusion may not transfer to social media or code-mixed text, where LLMs' broad pretraining could help more.
  • If the gap holds across domains, it suggests a general principle: for structured token-level tasks, task-specific heads on pretrained encoders are a strong default even in the LLM era.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper reports a comparative study of fine-tuned MahaBERT-v2 models against zero-shot general-purpose LLMs for low-resource Marathi NER. The authors construct three test-set variants from the MahaNER corpus (normal, normal-repeat-4x, random-repeat-4x) and evaluate five fine-tuned models (base and four augmentation variants) alongside four LLMs (Gemini, LLaMA-3.3-70B, Gemma-2-9B, Gemma-2-27B). The central claim is that fine-tuned MahaBERT models achieve F1 scores of 0.88–0.91, outperforming the best LLM (Gemini) by more than 0.20 F1 points. The paper also investigates data augmentation via self-concatenation and random concatenation, concluding that matching augmentation strategies improve performance on corresponding test sets.

Significance. If the evaluation is unbiased, the result provides a clear and practically useful confirmation that task-specific fine-tuned encoders remain superior to zero-shot general-purpose LLMs for a low-resource NER task in Marathi. The paper is among the first systematic comparisons of this kind for Marathi and could inform practitioners building NER systems for low-resource languages. However, the central comparison is weakened by an unspecified token-alignment procedure for LLM outputs, and the augmentation analysis is partly self-referential because test sets are built with the same operations used for training augmentation. The paper does not supply code or detailed hyperparameters, limiting reproducibility.

major comments (4)
  1. [§5.2, §7.3] The only description of how LLM outputs were matched to gold tokens is "Token alignment was enforced" (Section 5.2). Given Section 7.3 acknowledges that LLMs may "skip tokens, merge them, or produce labels in inconsistent formats," the alignment algorithm is load-bearing for the reported LLM scores in Tables 3–5. Without specifying whether unmatched tokens are discarded, whether fuzzy span matching is allowed, or whether reordering is permitted, the scores are not reproducible and could be systematically biased. Please provide the exact alignment routine or code, and ideally report raw and aligned scores.
  2. [§3.2, §3.3, §7.2] The test-set variants (Normal Repeat 4x and Random Repeat 4x) are constructed using the same repetition and concatenation operations as the augmented training sets in Section 3.3. The finding that augmented models perform best on the matching test set (Sections 6.2–6.3, discussed in 7.2) may therefore reflect train/test distribution matching rather than improved robustness or generalization. To support the augmentation claim, evaluate on independent test sets (e.g., naturally long sentences or a held-out domain) or include a control using the base model on augmented test sets.
  3. [§4.1, Tables 3–5] Fine-tuning hyperparameters (learning rate, batch size, number of epochs, maximum sequence length, optimizer, random seeds) are not reported; Section 4.1 only states "same hyperparameter settings." Additionally, no variance estimates, confidence intervals, or significance tests are provided for the 500-sentence evaluation. The F1 differences among fine-tuned variants are small (0.88–0.91); without repeated runs or statistical tests it is unclear whether the augmentation effects are meaningful. Report the hyperparameters and include error bars or significance testing.
  4. [§3.2, §5.2] The paper states that "All models were tested on exactly the same 500 Marathi sentences" but does not explain how these 500 sentences are selected from the MahaNER test split. If the Normal dataset is the full standard test split, its size should be given; if it is a subset, the selection procedure must be documented to ensure representativeness. This detail is essential for judging the evaluation's coverage and for reproducibility.
minor comments (5)
  1. [§6.1, §6.2] The "existing MahaNER baseline" scores (0.8843, 0.8668) are mentioned in the text but not included in any table or evaluated in the same protocol. Clarify whether these are the originally reported numbers and whether they were obtained under the same evaluation pipeline.
  2. [Table 2] The column "Objective" is vague (e.g., "Baseline," "Consistency," "Context diversity"). Provide concrete descriptions of what each test set measures and how it is derived from the original split.
  3. [References] Several references use incomplete author lists (e.g., "and 1 others") and contain typographical errors: "V olume" in Devlin et al., "F orum" in Chaudhari et al. Please correct these and follow a consistent citation style.
  4. [§7.2] The reference to "pretraining-style data diversity strategies (Liu et al., 2019; Raffel et al., 2020)" is not directly about fine-tuning data augmentation. Clarify the intended connection or cite more relevant augmentation literature.
  5. [§6.3] For the Random Repeat 4x dataset, "LLM performance drops notably, with LLaMA and Gemma models falling to F1 scores between 0.35 and 0.49" — but Gemini remains at 0.68. Specify which models and scores are being referred to for precision.

Circularity Check

1 steps flagged · score 4.0 of 10

Central BERT-vs-LLM comparison is independent, but the augmentation evaluation is self-referential: matched test sets make the reported augmentation benefit partly by construction.

  1. self definitional [Sections 3.2–3.3 vs Section 7.2 (Tables 4–5)]
    "Section 3.2: 'TheNormal Repeat 4xdataset is created by repeating each test sentence four times within a single sequence.' Section 3.3: 'multiple augmented training sets were created from the MahaNER training split using the same repetition strategies described above.' Section 7.2: 'models trained with augmented data generalise better to the corresponding augmented test sets.'"

    The test sets used to demonstrate the benefit of each augmentation are produced by the same construction operator (repeat 4x, or random-concat 4x) that defines the training condition. A model trained on R(train) is evaluated on R(test), where R is identical; the 'corresponding' test set is therefore in-distribution for the augmented model and out-of-distribution for the base model by design. The reported pattern—normal-repeat models excel on Normal Repeat 4x, random-repeat model excels on Random Repeat 4x—is baked into the experimental setup rather than being an independent test of generalization to genuinely new input structures. This makes the augmentation 'prediction' equivalent, up to empirical noise, to matching the training and test distributions by construction.

full rationale

The paper's main claim—fine-tuned MahaBERT-v2 models outperform zero-shot LLMs on Marathi NER by more than 0.20 F1—is an independent empirical measurement. The fine-tuned models are trained and evaluated on the MahaNER test sentences; the LLM numbers come from the same 500-sentence test set. No parameter is fit to the test set, and the MahaBERT/MahaNER resources, though from the same research group, are used as external artifacts rather than as authority for the result. The published MahaNER baseline (0.8843) is a secondary comparison. The one genuinely circular element is the augmentation analysis: the Normal Repeat 4x and Random Repeat 4x test sets are constructed with the very same concatenation/repetition operators used to build the augmented training sets, so the 'corresponding augmented test sets' in Section 7.2 are not independent probes. The finding that normal-repeat models do best on normal-repeat test data and random-repeat models do best on random-repeat test data is largely a consequence of matching training and test distributions. This does not undermine the head-to-head LLM comparison, but it does mean the paper's data-augmentation 'prediction' is partly self-definitional. Separately, Section 5.2's unspecified 'token alignment' procedure is a load-bearing methodological omission for the LLM comparison; it is a correctness/reproducibility risk, not a circularity. Section 3.3's statement that 'all test sets remain unmodified' also conflicts with Section 3.2's construction of modified test variants, reinforcing that the augmentation test sets are not independent. Overall, the central claim survives; the partial circularity is confined to the secondary augmentation evaluation, giving score 4.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new scientific entities. Its load-bearing assumptions are about dataset validity, test-split independence, LLM output alignment, and the tagging scheme. Free parameters are the chosen augmentation multipliers, unreported fine-tuning hyperparameters, and the arbitrary 500-sentence test subset.

free parameters (3)
  • Augmentation repetition factors = 4x, 10x
    Two hand-selected repetition multipliers for self-concatenation and random concatenation; no sweep or justification for these specific values (Section 3.3).
  • Fine-tuning hyperparameters
    Learning rate, epochs, batch size, and random seeds are not reported; all variants are said to share settings but the settings are absent (Section 4.1).
  • Test subset size = 500 sentences
    Evaluation uses 500 Marathi sentences; how this subset was selected from MahaNER is not specified (Section 5.2).
assumptions (4)
  • domain assumption MahaNER annotations are correct and representative of Marathi news text.
    The entire evaluation treats MahaNER as ground truth (Section 3.1).
  • domain assumption The 500-sentence test subset is independent of the fine-tuning training split.
    No data-leakage check is reported; the paper only states all models were tested on the same 500 sentences (Section 5.2).
  • domain assumption Token alignment for LLM outputs maps predictions to gold tokens without systematic bias.
    The paper asserts token alignment was enforced but does not describe the algorithm (Section 5.2).
  • domain assumption Non-IOB label scheme is appropriate for token-level F1 comparison.
    The paper uses non-IOB tags and notes this may differ from IOB2/BIOES results (Section 3.1, Limitations).

how reviews work

0 comments
Cite this review

Pith. "Pith review of BERT-based Models vs. Large Language Models for Low-Resource Named Entity Recognition: A Comparative Study on Marathi." pith.science (2026). https://pith.science/paper/VQNB6G3C

@misc{pith2026260723344,
  author       = {Pith},
  title        = {Pith review of: BERT-based Models vs. Large Language Models for Low-Resource Named Entity Recognition: A Comparative Study on Marathi},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VQNB6G3C}},
  note         = {Machine review of arXiv:2607.23344}
}
read the original abstract

Named Entity Recognition (NER) for low-resource languages such as Marathi remains a challenging task due to limited annotated resources and linguistic complexity. Although recent Large Language Models (LLMs) have demonstrated strong performance across a wide range of natural language processing tasks, their effectiveness for language-specific NER in low-resource settings remains uncertain. In this study, we fine-tune MahaBERT-v2 on different variants of the MahaNER dataset and systematically compare the performance of these models with an existing MahaNER baseline and prominent general-purpose LLMs, including Gemini, LLaMA-3.3-70B, and Gemma models. All models are evaluated on a Marathi NER test dataset using standard metrics of precision, recall, and F1-score. The experimental results show that the fine-tuned MahaBERT-based models consistently outperform both the baseline and all evaluated LLMs, with the fine-tuned models achieving F1-scores ranging from 0.88 to 0.91, surpassing the existing MahaNER model (0.8843) and significantly exceeding the performance of LLM-based approaches, whose F1-scores range from 0.57 to 0.69. These findings demonstrate that task-specific, language-focused models trained on domain-relevant data remain more effective than general-purpose LLMs for Marathi NER, highlighting the continued importance of specialized architectures for low-resource language processing.

Figures

Figures reproduced from arXiv: 2607.23344 by the authors.

Figure 1
Figure 1. Overview of the evaluation pipeline. Both fine-tuned MahaBERT models and zero-shot LLMs are tested [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Illustration of the two augmentation strategies. Self-concatenation (SC) repeats the same sentence multiple [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Architecture of the fine-tuned MahaBERT-v2 NER model. The self-attention mechanism of the transformer [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

16 extracted references · 4 linked inside Pith

  1. [1]

    Jacob Devlin and Ming-Wei Chang and Kenton Lee and Kristina Toutanova , title =. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long and Short Papers) , pages =. 2019 , publisher =

    2019

  2. [2]

    Proceedings of the WILDRE-6 workshop within the 13th language resources and evaluation conference , pages=

    L3cube-mahacorpus and mahabert: Marathi monolingual corpus, marathi bert language models, and resources , author=. Proceedings of the WILDRE-6 workshop within the 13th language resources and evaluation conference , pages=

  3. [3]

    Proceedings of the WILDRE-6 Workshop within the 13th Language Resources and Evaluation Conference , pages=

    L3cube-mahaner: A marathi named entity recognition dataset and bert models , author=. Proceedings of the WILDRE-6 Workshop within the 13th Language Resources and Evaluation Conference , pages=

  4. [4]

    Proceedings of the 15th Annual Meeting of the Forum for Information Retrieval Evaluation , pages=

    L3Cube-MahaSocialNER: A Social Media based Marathi Named Entity Recognition Dataset and BERT models , author=. Proceedings of the 15th Annual Meeting of the Forum for Information Retrieval Evaluation , pages=

  5. [5]

    Gomez and

    Ashish Vaswani and Noam Shazeer and Niki Parmar and Jakob Uszkoreit and Llion Jones and Aidan N. Gomez and. Attention Is All You Need , booktitle =

  6. [6]

    Unsupervised Cross-lingual Representation Learning at Scale , booktitle =

    Alexis Conneau and Kartikay Khandelwal and Naman Goyal and Vishrav Chaudhary and Guillaume Wenzek and Francisco Guzm. Unsupervised Cross-lingual Representation Learning at Scale , booktitle =. 2020 , publisher =

    2020

  7. [7]

    Abhimanyu Dubey and Abhinav Jauhri and Abhinav Pandey and Abhishek Kadian and Ahmad Al. The. arXiv preprint arXiv:2407.21783 , year =

    arXiv

  8. [8]

    arXiv preprint arXiv:2403.08295 , year =

    Gemma: Open Models Based on. arXiv preprint arXiv:2403.08295 , year =

    arXiv

Show all 16 references
  1. [9]

    Tom B. Brown and Benjamin Mann and Nick Ryder and Melanie Subbiah and Jared Kaplan and Prafulla Dhariwal and Arvind Neelakantan and Pranav Shyam and Girish Sastry and Amanda Askell and Sandhini Agarwal and others , title =. Advances in Neural Information Processing Systems 33 (

  2. [10]

    Chi and Quoc V

    Jason Wei and Xuezhi Wang and Dale Schuurmans and Maarten Bosma and Brian Ichter and Fei Xia and Ed H. Chi and Quoc V. Le and Denny Zhou , title =. Advances in Neural Information Processing Systems 35 (

  3. [11]

    arXiv preprint arXiv:1907.11692 , year =

    Yinhan Liu and Myle Ott and Naman Goyal and Jingfei Du and Mandar Joshi and Danqi Chen and Omer Levy and Mike Lewis and Luke Zettlemoyer and Veselin Stoyanov , title =. arXiv preprint arXiv:1907.11692 , year =

    1907 arXiv

  4. [12]

    2022 , doi =

    Jing Li and Aixin Sun and Jianglei Han and Chenliang Li , title =. 2022 , doi =

    2022

  5. [13]

    Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages =

    Guillaume Lample and Miguel Ballesteros and Sandeep Subramanian and Kazuya Kawakami and Chris Dyer , title =. Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , pages =. 2016 , publisher =

    2016

  6. [14]

    Liu , title =

    Colin Raffel and Noam Shazeer and Adam Roberts and Katherine Lee and Sharan Narang and Michael Matena and Yanqi Zhou and Wei Li and Peter J. Liu , title =. Journal of Machine Learning Research , volume =. 2020 , url =

    2020

  7. [15]

    Tjong Kim Sang and Fien De Meulder , title =

    Erik F. Tjong Kim Sang and Fien De Meulder , title =. Proceedings of the Seventh Conference on Natural Language Learning at. 2003 , publisher =

    2003

  8. [16]

    arXiv preprint arXiv:2205.14728 , year=

    L3cube-mahanlp: Marathi natural language processing datasets, models, and library , author=. arXiv preprint arXiv:2205.14728 , year=

    arXiv

Pith tools

Reviewed July 31, 2026 · model on record in the stance chip above.