REVIEW 4 major objections 5 minor 21 references
L3Cube-MahaSTS: A Marathi Sentence Similarity Dataset and Models
T0 review · 4 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read This paper introduces MahaSTS, the first human-annotated Marathi sentence-similarity dataset, and shows that fine-tuning MahaSBERT on it reaches Pearson 0.9600 and Spearman 0.9523, beating multilingual baselines.
desk verdict MahaSTS is a valuable new Marathi STS dataset, but the reported 0.96 correlations are likely inflated by a closed data-construction and evaluation loop. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is the pairing of a balanced dataset-construction pipeline with Sentence-BERT fine-tuning. Sentence-BERT is a siamese BERT architecture that maps each sentence to a fixed embedding and is trained so that embedding cosine similarity matches the human label; here the model is MahaSBERT, a Marathi SBERT already trained on NLI data. The dataset pipeline selects candidate pairs from one million real Marathi sentences using MahaSBERT-STS cosine similarity, places them into six non-overlapping score buckets, and keeps exactly 2,810 pairs per bucket after human annotation. That uniform bucketing is what lets the regression training see the full 0-5 range in equal proportion, which the paper argues reduces label bias and stabilizes learned similarity scores.
What would settle it
Sample 1,000 Marathi sentence pairs at random from MahaCorpus without prefiltering by cosine similarity, have fresh annotators score them 0-5 without seeing model outputs, and compute MahaSBERT-STS-v2's Pearson and Spearman correlations against those labels; if the correlations are materially below 0.96 and 0.95, the construction pipeline was the source of the reported agreement.
Extended reading notes
Core claim
On its own terms, the paper establishes that a Marathi STS dataset built from real sentences rather than machine translation can support effective similarity training. MahaSTS contains 16,860 pairs uniformly spread over six similarity buckets from 0 to 5, and MahaSBERT-STS-v2 — MahaSBERT fine-tuned on the training split with CosineSimilarityLoss, MEAN pooling, two epochs, and a 1e-5 learning rate — attains Pearson 0.9600 and Spearman 0.9523 on the held-out test split. This beats every baseline in the comparison, including the same base model before STS fine-tuning and strong multilingual models. The paper also reports that MEAN pooling outperforms CLS and MAX pooling, and acknowledges a limitation on longer or more complex Marathi sentences.
Load-bearing premise
The reported correlations assume that sentence pairs selected and bucketed by MahaSBERT-STS cosine similarity, then labeled with the model's scores as a reference, represent how Marathi speakers judge similarity in general; if that selection pipeline makes the test set easy or non-representative, the 0.95 numbers overstate real-world performance.
Editorial extensions
If this is right
- Marathi NLP gains a reusable STS benchmark and a ready fine-tuned model, so retrieval, question answering, clustering, and paraphrase detection can be built and evaluated on native Marathi text rather than translated English data.
- The paper argues that the balanced six-bucket design reduces label bias and stabilizes regression training, so the resulting model is usable across the full 0-5 range, from unrelated pairs to near-equivalents.
- The comparison shows task-specific fine-tuning on human-annotated data beating both general Marathi BERT and multilingual sentence encoders, strengthening the case that annotation quality, not model scale, is the bottleneck in low-resource STS.
- The released dataset and model give other researchers a fixed benchmark and baseline, making future Marathi sentence-similarity results directly comparable.
Reading between the lines
- Editorial inference: since candidate pairs were selected and bucketed using MahaSBERT-STS cosine similarity, with those scores also used as annotation reference, an independent sample of naturally occurring Marathi sentence pairs could yield lower correlations than 0.96/0.95; the reported numbers are conditioned on the selection pipeline.
- Editorial inference: applying the same bucket-balanced annotation recipe to other low-resource languages would be a direct test of the paper's recipe, and the decisive check is whether such models improve downstream retrieval or paraphrase tasks beyond benchmark correlation.
- Editorial inference: the uniform bucket distribution makes the test set much flatter than real similarity distributions, where most random pairs are dissimilar, so Spearman gains may partly reflect stratified sampling; reporting performance on an unstratified sample would clarify real-world ranking quality.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces MahaSTS, a Marathi sentence-textual-similarity dataset of 16,860 sentence pairs with continuous 0–5 similarity labels, uniformly distributed across six score buckets. The pairs are drawn from L3Cube-MahaCorpus by using MahaSBERT-STS cosine similarity to select one candidate sentence per query per bucket, with annotators using those cosine scores as a reference. The authors fine-tune MahaSBERT with CosineSimilarityLoss on an 85% train split and report Pearson 0.9600 and Spearman 0.9523 on the 10% test split, beating several multilingual and Marathi baselines. They release the dataset and the fine-tuned model for public use.
Significance. If the reported results are valid, MahaSTS would be a valuable low-resource resource: it is, to my knowledge, the first human-annotated Marathi STS dataset built from naturally occurring text, it is large (16,860 pairs), and it is deliberately balanced across the similarity range. The paper also ships the dataset and model publicly, which is a strength that makes external scrutiny and reuse possible. However, the significance is currently conditional. The evaluation is conducted entirely on a test split of the same dataset whose pair selection and bucket assignment were produced by MahaSBERT-STS, and whose labels were anchored to that model's cosine scores. As a result, the headline correlations and the baseline ranking in Table 3 do not currently establish that the model generalizes to Marathi sentence similarity as judged by humans in general. The core contribution is still plausible, but it needs external validation and more transparent annotation reporting.
major comments (4)
- [Section 3.1–3.2, Table 3] The test set is not an independent sample of Marathi similarity judgments. Candidate pairs were selected by retrieving, for each query sentence, candidates whose MahaSBERT-STS cosine similarity fell into fixed intervals, and the annotators were given those cosine scores 'as a reference' during labeling. The 10% test split is drawn from this same curated pool. Consequently, the reported Pearson 0.9600 and Spearman 0.9523, and the relative ordering of baselines in Table 3, are measured on a distribution whose composition and bucket boundaries were generated by the same model family that is being evaluated. This makes the evaluation self-referential, and I do not think the current results establish generalization to real-world Marathi semantic similarity.
- [Section 5, Results and Discussion] External validation is needed to support the central claim that MahaSTS 'enables effective training' for STS. I recommend evaluating on at least one held-out resource not produced by the MahaSBERT-STS selection pipeline, for example independently sampled sentence pairs from MahaCorpus, a Marathi translation of an existing STS benchmark, or the MahaParaphrase corpus. Without such a check, the improvement claimed in Section 5 could be largely an artifact of learning the selector's bucket structure rather than of genuinely improved semantic similarity modeling.
- [Section 3.2, Dataset Curation] The dataset's reliability depends on annotation quality, but the paper reports no annotator count, no inter-annotator agreement, and no detailed annotation protocol. It also does not specify how the completely dissimilar pairs in bucket 0 were generated. Since the paper's main contribution is a human-annotated, balanced dataset, these details are load-bearing; at minimum, report IAA on a subset and document the full annotation instructions, including how the reference cosine scores were presented to annotators.
- [Section 6, Limitations] The Limitations section only discusses sentence length and complexity. It should explicitly acknowledge the selection and anchoring bias: candidate pairs were chosen by MahaSBERT-STS cosine similarity, the buckets were defined by that model's score intervals, and annotators saw those scores as references. This is important for users of the dataset to interpret its coverage and for future work to design less circular evaluation protocols.
minor comments (5)
- [Section 3.2] The construction of bucket 0 is not described beyond saying that completely dissimilar pairs were 'eventually put' into it; please specify the selection procedure for these pairs.
- [Section 3.2] The bucket boundaries (0.8, 1], (0.6, 0.8], etc. are defined on the embedding model's cosine scale, yet the final labels are on a 0–5 semantic scale; the relationship between these two scales should be clarified and justified.
- [Section 4.2] The description of MahaBERT is confusing: the text says it is a multilingual BERT fine-tuned on Marathi data, while elsewhere it is described as a Marathi monolingual model; please clarify the base architecture and training corpus.
- [Section 5, Table 3] The text refers to models 'described in Section 3,' but the model descriptions are actually in Section 4; also, using HuggingFace model IDs in the table makes the table hard to read without the repository links.
- [Section 3.1] MahaSBERT-STS is only given as a hyperlink; please provide a formal citation, model version, and a short technical description so the selection process is reproducible.
Circularity Check
Reported 0.96 correlations are measured on a test set whose candidate selection and label reference are produced by the same MahaSBERT-STS model family that is later fine-tuned; the headline performance claim is substantially self-referential.
-
fitted input called prediction
[Sections 3.1-3.2 (data collection and preprocessing); evaluation in Section 5 (Table 3)]
"We created embeddings of all sentences in the file using MahaSBERT-STS. We picked 5000 random sentences from the 1M corpus as query sentences. We then calculated the cosine similarity between each query sentence and the 1M corpus sentences. ... we started annotating the sentences, using the cosine similarity scores from MahaSBERT-STS as a reference to help us with the annotations."
The same model family (L3Cube/MahaSBERT, specifically MahaSBERT-STS) selects the candidate pairs, defines the cosine-similarity buckets from which the train/test splits are drawn, and is provided to annotators as a labeling reference. MahaSBERT-STS-v2 is then fine-tuned from the same family and evaluated on the held-out slice of that self-constructed pool. Consequently the Pearson 0.9600 / Spearman 0.9523 results largely reflect consistency between the fine-tuned model and its parent model's ranking rather than an independent human-judgment benchmark; no external STS test set is used, so the central accuracy claim reduces, partially, to a within-family self-consistency check.
full rationale
Only one circular step is load-bearing, and it is partial rather than total. The human annotators and held-out split supply some independent signal: the labels are not literally the output of MahaSBERT-STS, and the test pairs are not scored by that model at test time. However, the construction loop is explicit: MahaSBERT-STS determines which pairs are included, in which bucket they sit, and the reference scale shown to annotators. MahaSBERT-STS itself is not defined or evaluated in the paper; it is only a Hugging Face model link. Because MahaSBERT-STS-v2 belongs to the same L3Cube/MahaSBERT family, the reported correlations are not evidence of general Marathi sentence-similarity ability; they are evidence that the fine-tuned model can reproduce the selector's notion of similarity. The paper's Limitations section only mentions sentence-length degradation and does not address this selection/anchoring bias. The baselines in Table 3 are compared on the same self-constructed test set, so the ranking is also confounded. I score 6 rather than 8-10 because independent human judgment and a real train/test split prevent the derivation from being fully definitional, but the central performance claim is substantially contaminated by the data-construction pipeline.
Assumptions & free parameters
free parameters (1)
- cosine similarity bucket boundaries for data selection =
(0,0.2], (0.2,0.4], (0.4,0.6], (0.6,0.8], (0.8,1]
assumptions (3)
- domain assumption Cosine similarity from MahaSBERT-STS is a valid proxy for semantic relatedness when selecting and bucketing sentence pairs.
- domain assumption Uniform distribution across six score buckets reduces label bias and improves regression training stability.
- domain assumption Human annotations are reliable without reported inter-annotator agreement or annotator guidelines.
Cite this review
Pith. "Pith review of L3Cube-MahaSTS: A Marathi Sentence Similarity Dataset and Models." pith.science (2026). https://pith.science/paper/VPPAMEEA
@misc{pith2026250821569,
author = {Pith},
title = {Pith review of: L3Cube-MahaSTS: A Marathi Sentence Similarity Dataset and Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/VPPAMEEA}},
note = {Machine review of arXiv:2508.21569}
}
read the original abstract
We present MahaSTS, a human-annotated Sentence Textual Similarity (STS) dataset for Marathi, along with MahaSBERT-STS-v2, a fine-tuned Sentence-BERT model optimized for regression-based similarity scoring. The MahaSTS dataset consists of 16,860 Marathi sentence pairs labeled with continuous similarity scores in the range of 0-5. To ensure balanced supervision, the dataset is uniformly distributed across six score-based buckets spanning the full 0-5 range, thus reducing label bias and enhancing model stability. We fine-tune the MahaSBERT model on this dataset and benchmark its performance against other alternatives like MahaBERT, MuRIL, IndicBERT, and IndicSBERT. Our experiments demonstrate that MahaSTS enables effective training for sentence similarity tasks in Marathi, highlighting the impact of human-curated annotations, targeted fine-tuning, and structured supervision in low-resource settings. The dataset and model are publicly shared at https://github.com/l3cube-pune/MarathiNLP
Figures
Reference graph
Works this paper leans on
-
[1]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
Eneko Agirre, Daniel Cer, Mona Diab, and Aitor Gonzalez-Agirre. 2012. https://aclanthology.org/S12-1051 S em E val-2012 task 6: A pilot on semantic textual similarity . In Proceedings of the Sixth International Workshop on Semantic Evaluation ( S em E val 2012) , pages 385--393, Montr \'e al, Canada. Association for Computational Linguistics
work page 2012
-
[4]
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity-multilingual and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055
arXiv 2017
-
[5]
Alexis Conneau, Douwe Kiela, Holger Schwenk, Lo \" c Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. arXiv preprint arXiv:1705.02364
arXiv 2017
-
[6]
Ishita Dasgupta, Demi Guo, Andreas Stuhlm \"u ller, Samuel J Gershman, and Noah D Goodman. 2018. Evaluating compositionality in sentence embeddings. arXiv preprint arXiv:1802.04302
arXiv 2018
-
[7]
Samruddhi Deode, Janhavi Gadre, Aditi Kajale, Ananya Joshi, and Raviraj Joshi. 2023. L3cube-indicsbert: A simple approach for learning cross-lingual sentence representations using multilingual bert. In Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation, pages 154--163
2023
-
[8]
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171--4186
2019
Show all 21 references
-
[9]
Bill Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Third international workshop on paraphrasing (IWP2005)
2005
-
[10]
Fangxiaoyu Feng, Yinfei Yang, Daniel Cer, Naveen Arivazhagan, and Wei Wang. 2020. Language-agnostic bert sentence embedding. arXiv preprint arXiv:2007.01852
2020 arXiv
-
[11]
Suramya Jadhav, Abhay Shanbhag, Amogh Thakurdesai, Ridhima Sinare, Ananya Joshi, and Raviraj Joshi. 2025. Mahaparaphrase: A marathi paraphrase detection corpus and bert-based models. arXiv preprint arXiv:2508.17444
2025 arXiv
-
[12]
Ananya Joshi, Aditi Kajale, Janhavi Gadre, Samruddhi Deode, and Raviraj Joshi. 2023. L3cube-mahasbert and hindsbert: Sentence bert models and benchmarking bert sentence representations for hindi and marathi. In Science and Information Conference, pages 1184--1199. Springer
2023
-
[13]
Raviraj Joshi. 2022. L3cube-mahacorpus and mahabert: Marathi monolingual corpus, marathi bert language models, and resources. In Proceedings of the WILDRE-6 Workshop within the 13th Language Resources and Evaluation Conference, pages 97--101
2022
-
[14]
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)...
2020
-
[15]
u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, and 1 others. 2020. Retrieval-augmented generation for knowledge-intensive nlp tasks. Advances in neural informati...
2020
-
[16]
Bohan Li, Hao Zhou, Junxian He, Mingxuan Wang, Yiming Yang, and Lei Li. 2020. On the sentence embeddings from pre-trained language models. arXiv preprint arXiv:2011.05864
2020 arXiv
-
[17]
Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJC...
2019
-
[18]
Raphael Scheible, Fabian Thomczyk, Patric Tippmann, Victor Jaravine, and Martin Boeker. 2020. Gottbert: a pure german language model. arXiv preprint arXiv:2012.02110
2020 arXiv
-
[19]
Milan Straka, Jakub N \'a plava, Jana Strakov \'a , and David Samuel. 2021. Robeczech: Czech roberta, a monolingual contextualized language representation model. In International conference on text, speech, and dialogue, pages 197--209. Springer
2021
-
[20]
Xin Tang, Shanbo Cheng, Loc Do, Zhiyu Min, Feng Ji, Heng Yu, Ji Zhang, and Haiqin Chen. 2018. Improving multilingual semantic textual similarity with shared sentence encoder for low-resource languages. arXiv preprint arXiv:1810.08740
2018 arXiv
-
[21]
Abhishek Velankar, Hrushikesh Patil, and Raviraj Joshi. 2022. Mono vs multilingual bert for hate speech detection and text classification: A case study in marathi. In IAPR workshop on artificial neural networks in pattern recognition, pages 121--128. Springer
2022
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.