REVIEW 4 major objections 6 minor 26 references
NLPineers@ NLU of Devanagari Script Languages 2025: Hate Speech Detection using Ensembling of BERT-based models
T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read A fixed cascade of three fine-tuned multilingual BERT models reaches 0.7762 recall and 0.6914 F1 on the Devanagari hate-speech test set, improving on every single model it is compared with.
desk verdict A modest, honest shared-task system description whose headline ensemble result is partly a test-set selection artifact, but still a useful external benchmark for Devanagari hate speech. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the cascade decision function: output hate if the XLM-RoBERTa model (M7) outputs hate; otherwise output hate if the MuRIL model (M3) outputs hate; otherwise output the decision of the MuRIL-abusive model (M1). It is a deterministic, OR-like composition of three fine-tuned classifiers with no joint training. Each stage is a transformer with its native classification head: M7 is chosen for highest single-model recall, M3 for balanced F1, and M1 for highest precision and accuracy. The cascade works by letting the recall-oriented model vote first, the balanced model catch remaining positives, and the precision-oriented model act as a conservative fallback. This ordering carries the argument because it turns three individually weaker models into a single predictor with higher recall than any of them.
What would settle it
Have native Hindi and Nepali speakers annotate a sample of the augmented examples that passed the 0.9 cosine filter, and re-run the pipeline without augmentation. If a substantial share of the filtered examples are no longer hate speech, or if the recall gain disappears without augmentation, then the ensemble's improvement is an artifact of duplicated near-identical examples rather than better detection.
Extended reading notes
Core claim
The central claim is that a three-model cascade, not a learned ensemble but a fixed decision rule, detects hate speech in Devanagari-scripted Hindi and Nepali tweets better than any of the eight individual classifiers evaluated, especially in recall. The rule is: output hate if model M7, an XLM-RoBERTa model, outputs hate; otherwise output hate if model M3, a MuRIL model, outputs hate; otherwise output the decision of model M1, a MuRIL model fine-tuned on abusive-language data. On the official test set the cascade scores 0.7762 recall, 0.6639 precision, 0.6914 F1, and 0.8258 accuracy, placing third of thirty-one by recall and seventeenth by F1. The single best recall model, M7, reaches 0.7381 recall and 0.6933 F1, while the most accurate model, M1, reaches 0.8950 accuracy; the cascade trades a little precision and accuracy for a large recall gain. The authors attribute the improvement to complementary strengths: one model catches the most positives, a second gives balanced coverage, and a high-precision fallback limits false positives when the first two abstain.
Load-bearing premise
The load-bearing premise is that the backtranslated and duplicated hate examples, filtered by cosine similarity above 0.9, still carry their original hate label and teach the model real hate speech rather than translation artifacts; the authors themselves concede that backtranslation can shift respectful Nepali terms to informal register and change sentiment.
Editorial extensions
If this is right
- For Devanagari Hindi and Nepali hate speech, fine-tuned multilingual BERT models can be combined by a fixed cascade without any ensemble training, which is a cheap recipe for low-resource classification tasks.
- The recall gain from 0.7381 to 0.7762 means roughly four more hate instances per hundred are caught by the ensemble, at the cost of slightly more non-hate tweets being flagged as hate.
- XLM-RoBERTa is the strongest single model for recall on this mixed-script dataset, while IndicBERT is weaker, suggesting Nepali text is the bottleneck for Indic-specific models.
- Backtranslation with a cosine-similarity filter above 0.9, plus duplication of hate instances, is enough to train competitive transformers despite only 2,214 original hate examples in the training set.
Reading between the lines
- The cascade's decision rule is equivalent to an OR over the first two models, so alternative orderings, such as putting the high-precision model first, would likely change the precision-recall tradeoff; the paper does not test them.
- A direct annotation of backtranslated instances that passed the 0.9 cosine filter would separate the ensemble's genuine linguistic gain from memorization of duplicated examples, and the threshold could be swept to find where label preservation breaks.
- Because Hindi and Nepali share the Devanagari script but differ in vocabulary, the language-agnostic embeddings may confuse near-identical words; conditioning the cascade on a language-identification head is a natural extension the paper leaves open.
- If language labels for the test set become available, the recall gain could be decomposed per language, since the paper reports only pooled scores and we cannot tell whether the cascade helps Nepali, Hindi, or both.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper describes a hate speech detection system for Devanagari-scripted Hindi and Nepali, developed for Subtask B of the CHIPSAL@COLING 2025 shared task. The authors fine-tune eight models (MuRIL, XLM-RoBERTa, IndicBERT, and FastText-based variants), apply backtranslation with cosine-similarity filtering for data augmentation, and combine three models (M7, M3, M1) into a priority-based cascade. On the official test set they report recall 0.7762 (rank 3/31), precision 0.6639, F1 0.6914 (rank 17/31), and accuracy 0.8258. The central claim is that this ensemble improves recall over any single model for low-resource Devanagari hate speech detection.
Significance. If the reported results are taken at face value, the paper is a useful system description for a shared task: it compares eight architectures on a low-resource Devanagari hate speech task, uses an official external leaderboard for evaluation, and makes code publicly available. The finding that a multilingual BERT cascade achieves high recall (rank 3 of 31) is of practical interest for recall-oriented moderation scenarios. However, the central claim that the ensemble 'demonstrates' improved recall is weakened by two load-bearing issues: the ensemble members and ordering are selected using the test-set metrics in Table 3, and the cascade rule is logically an OR of three models, so part of the recall gain is guaranteed by construction. The paper also contains a numerical inconsistency in the augmentation accounting and no uncertainty estimates. With a validation-based selection protocol and reframed claims, the contribution would be solid.
major comments (4)
- [Section 4.3, Table 3] The ensemble construction is selected on the test set. Section 4.3 states that M7, M3, and M1 were chosen as primary, secondary, and fallback 'based on each model's unique strengths' using Table 3, which reports test-set results. This means the reported ensemble metrics are optimistically biased relative to any pre-specified rule, because the model ordering was chosen after seeing the test labels. Moreover, the cascade rule is equivalent to predicting hate if any of M7, M3, or M1 predicts hate, so the ensemble recall is at least the maximum individual recall by construction; the increase from 0.7381 to 0.7762 is not by itself evidence that ensembling adds signal beyond taking the union of positive predictions. The paper should select the ensemble on a held-out development split and report the corresponding test metrics, and should compare the cascade against a plain OR of the three models or against M7 alone. Note also that Table 3 shows the ensemble F1 (0.6914) is slightly lower than M7's F1 (0.6933), so calling the ensemble the 'best performing model' in the abstract is at best selective.
- [Section 4.1, Table 1] The data augmentation accounting is internally inconsistent. Table 1 lists 2,214 training hate instances (679 Hindi + 1,535 Nepali) and 474 evaluation hate instances (142 Hindi + 332 Nepali). The text states that 'the training set grew to 13,695 instances by incorporating the original 2,214 training, 474 evaluation, and their augmented instances.' Starting from the Table 1 training total of 19,019, no combination of duplication and backtranslation of the hate subset yields 13,695. The paper should give exact counts of original, duplicated, and backtranslated examples per language, and clarify whether the 13,695 figure includes the non-hate training instances. As written, the data setup is not reproducible.
- [Section 4.4, Table 3] No repeated runs, seeds, or confidence intervals are reported; all model comparisons are point estimates on the test set. Given the small hate class (142 Hindi and 332 Nepali evaluation instances in Table 1), differences such as M7 recall 0.7381 versus M3 recall 0.6877 could be within noise. The paper should provide variance estimates from multiple fine-tuning runs on a development set, or at minimum explicitly state that the observed ordering is not statistically evaluated. Currently the model ranking in Table 3 is asserted without uncertainty quantification.
- [Section 4.1, Limitations] The augmentation pipeline assumes that backtranslated examples filtered by cosine similarity greater than 0.9 retain their hate labels. The Limitations section explicitly concedes that backtranslation can shift formal or respectful Nepali terms to informal registers and alter sentiment, so the cosine threshold does not guarantee label preservation. Because these augmented examples are used to train all models, the measured performance could reflect artifacts of the augmentation procedure rather than hate detection ability. The paper should add a small manual or automatic label-preservation check on the augmented samples, and should describe how the 0.9 threshold was chosen without reference to the test-set results.
minor comments (6)
- [Introduction] The text says 'The First Workshop on South East Asian Language Processing' but the cited workshop (Sarveswaran et al., 2025) is on South Asian languages; the region should be corrected.
- [References] References 7 and 8 (Conneau et al., 2019 and 2020) appear to be the same paper; they should be merged or clearly distinguished.
- [Table 2] The phrase 'an token embedding-based model' should read 'a token embedding-based model.'
- [Section 4.3] The cascade rule could be written more transparently as an OR of M7, M3, and M1; the current formulation obscures the fact that the ensemble's positive set is the union of the three models' positive predictions.
- [Figure 1] Figure 1 is referenced in Section 3 but no figure content is visible in the manuscript; please ensure the workflow figure is included in the final version.
- [Table 3] The caption refers to dark green and dark red cells for best and worst performance, but the printed table is monochrome; add explicit markers such as boldface or asterisks so the best and worst values are identifiable.
Circularity Check
Ensemble selected from test-set metrics and its recall gain is guaranteed by the OR cascade.
-
fitted input called prediction
[Section 4.3 (Ensemble Strategy) and Table 3]
"Our ensemble strategy leveraged the strengths of our top-performing models from Table 3. We chose M7 (XLM-Roberta) as the primary model, M3 (MuRIL) as the secondary model, and M1 (MuRIL abusive) as the fallback model, based on each model’s unique strengths."
The membership and priority order of the ensemble were selected by reading Table 3, whose caption identifies the numbers as 'Evaluation results on test set'. The same Table 3 then reports the ensemble's own test recall (0.7762), F1 (0.6914), and accuracy (0.8258). This is selection on the evaluation target: the reported ensemble scores are the result of choosing among models already scored on the same test set, so they are optimistically biased and cannot be read as an independent prediction of the ensemble's benefit.
-
self definitional
[Section 4.3 prediction rule; Table 3 rows M7 and Ensemble]
"prediction(x) = 1 if M7(x) = 1, 1 if M7(x) = 0 and M3(x) = 1, M1(x) otherwise ... M7 XLM-Roberta 0.7381 ... Ensemble (M1, M3, M7) 0.7762"
The cascade's positive predictions are exactly the union of M7's, M3's, and M1's positive predictions, with M7 prioritized first. For any fixed labeled set, the true positives of the union are a superset of the true positives of M7, so recall(ensemble) >= recall(M7) = 0.7381 by definition. The reported 0.7762 is therefore not evidence that ensembling learned a better hate-speech boundary; part of the observed recall gain is a set-theoretic guarantee of ORing predictions, not an empirical achievement of the ensemble strategy.
full rationale
Step 1 is the main circularity: Section 4.3 says the ensemble members and their order were chosen from Table 3, whose caption labels the numbers as test-set results; the same table then reports the final ensemble as 'best performing'. That is test-set selection, so the headline 0.7762 recall is partly a fitted outcome rather than an independent evaluation of the ensemble idea. Step 2 shows the recall comparison is partly tautological: the cascade is an OR of M7, M3, and M1, so recall cannot drop below M7's recall; the improvement from 0.7381 to 0.7762 is guaranteed by the union operation, with only its magnitude depending on data. No load-bearing self-citation is present, and the underlying model outputs plus the official Codalab scores are external evidence. The cosine-similarity augmentation choice ('chosen empirically') and the limitations-section admission that backtranslation can shift respectful Nepali terms to informal registers and alter sentiment are correctness risks, not circularity. Overall, the exact test numbers are real, but the paper's interpretive claim that the ensemble 'effectively leverages' the models is partly constructed and partly trivial, so a moderate-to-substantial circularity score is warranted.
Assumptions & free parameters
free parameters (5)
- Cosine similarity threshold for backtranslation filtering =
0.9
- Hate class duplication factor =
all hate instances duplicated (2x)
- Learning rate =
2e-5
- Batch size =
16
- Ensemble priority order =
M7, then M3, then M1
assumptions (4)
- domain assumption Backtranslation with mBART followed by cosine similarity filtering preserves the hate-speech label with high probability when similarity is above 0.9.
- domain assumption The CHIPSAL dataset labels and official Codalab evaluation are accurate ground truth.
- domain assumption Cosine similarity between XLM-RoBERTa embeddings is a valid proxy for semantic and label preservation after backtranslation.
- standard math Standard transformer fine-tuning machinery is reliable as implemented in Hugging Face.
Cite this review
Pith. "Pith review of NLPineers@ NLU of Devanagari Script Languages 2025: Hate Speech Detection using Ensembling of BERT-based models." pith.science (2026). https://pith.science/paper/XCQCOVMI
@misc{pith2026241208163,
author = {Pith},
title = {Pith review of: NLPineers@ NLU of Devanagari Script Languages 2025: Hate Speech Detection using Ensembling of BERT-based models},
year = {2026},
howpublished = {\url{https://pith.science/paper/XCQCOVMI}},
note = {Machine review of arXiv:2412.08163}
}
read the original abstract
This paper explores hate speech detection in Devanagari-scripted languages, focusing on Hindi and Nepali, for Subtask B of the CHIPSAL@COLING 2025 Shared Task. Using a range of transformer-based models such as XLM-RoBERTa, MURIL, and IndicBERT, we examine their effectiveness in navigating the nuanced boundary between hate speech and free expression. Our best performing model, implemented as ensemble of multilingual BERT models achieve Recall of 0.7762 (Rank 3/31 in terms of recall) and F1 score of 0.6914 (Rank 17/31). To address class imbalance, we used backtranslation for data augmentation, and cosine similarity to preserve label consistency after augmentation. This work emphasizes the need for hate speech detection in Devanagari-scripted languages and presents a foundation for further research.
Figures
Reference graph
Works this paper leans on
-
[1]
Rahul Aralikatte, Miryam De Lhoneux, Anoop Kunchukuttan, and Anders S gaard. 2021. Itihasa: A large-scale corpus for sanskrit to english translation. In Proceedings of the 8th Workshop on Asian Translation (WAT2021), pages 191--197
2021
-
[2]
Arup Baruah, Kaushik Das, Ferdous Barbhuiya, and Kuntal Dey. 2020. https://aclanthology.org/2020.trac-1.12 Aggression identification in E nglish, H indi and B angla text using BERT , R o BERT a and SVM . In Proceedings of the Second Workshop on Trolling, Aggression and Cyberbullying, pages 76--82, Marseille, France. European Language Resources Association (ELRA)
work page 2020
-
[3]
Tommaso Caselli, Valerio Basile, Jelena Mitrovic, and Michael Granitzer. 2020. https://arxiv.org/abs/2010.12472 Hatebert: Retraining BERT for abusive language detection in english . CoRR, abs/2010.12472
arXiv 2020
-
[4]
Abhishek Chopra, Deepak Kumar Sharma, Aashna Jha, and Uttam Ghosh. 2023. A framework for online hate speech detection on code-mixed hindi-english text and hindi text in devanagari. ACM Transactions on Asian and Low-Resource Language Information Processing, 22(5):1--21
work page 2023
-
[6]
Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. https://arxiv.org/abs/1911.02116 Unsupervised cross-lingual representation learning at scale . Preprint, arXiv:1911.02116
arXiv 2020
-
[7]
Mithun Das, Somnath Banerjee, and Animesh Mukherjee. 2022. Data bootstrapping approaches to improve low resource abusive language detection for indic languages. arXiv preprint arXiv:2204.12543
arXiv 2022
-
[8]
Edouard Grave, Piotr Bojanowski, Prakhar Gupta, Armand Joulin, and Tomas Mikolov. 2018. Learning word vectors for 157 languages. In Proceedings of the International Conference on Language Resources and Evaluation (LREC 2018)
2018
-
[9]
Farhan Ahmad Jafri, Kritesh Rauniyar, Surendrabikram Thapa, Mohammad Aman Siddiqui, Matloob Khushi, and Usman Naseem. 2024. Chunav: Analyzing hindi hate speech and targeted groups in indian election discourse. ACM Transactions on Asian and Low-Resource Language Information Processing
2024
Show all 26 references
-
[10]
Farhan Ahmad Jafri, Mohammad Aman Siddiqui, Surendrabikram Thapa, Kritesh Rauniyar, Usman Naseem, and Imran Razzak. 2023. Uncovering political hate speech during indian election campaign: A new low-resource dataset and baselines
2023
-
[11]
Khapra, and Pratyush Kumar
Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Khapra, and Pratyush Kumar. 2020. IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages . In Findings of EMNLP
2020
-
[13]
Talukdar
Simran Khanuja, Diksha Bansal, Sarvesh Mehtani, Savya Khosla, Atreyee Dey, Balaji Gopalan, Dilip Kumar Margam, Pooja Aggarwal, Rajiv Teja Nagipogu, Shachi Dave, Shruti Gupta, Subhash Chandra Bose Gali, Vish Subramanian, and Partha P. Talukdar. 2021 b . https://arxiv.org/abs/21...
2021 arXiv
-
[14]
Atharva Kulkarni, Meet Mandhane, Manali Likhitkar, Gayatri Kshirsagar, and Raviraj Joshi. 2021. L3cubemahasent: A marathi tweet-based sentiment analysis dataset. In Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Ana...
2021
-
[15]
Nishant Luitel, Nirajan Bekoju, Anand Kumar Sah, and Subarna Shakya. 2024. Can perplexity predict fine-tuning performance? an investigation of tokenization effects on sequential language models for nepali. arXiv preprint arXiv:2404.18071
2024 arXiv
-
[16]
Thomas Mandl, Sandip Modha, Marcos Zampieri, et al. 2020. Hasoc 2020: Hate speech and offensive content identification in indo-european languages. In Proceedings of the 12th Language Resources and Evaluation Conference (LREC), pages 190--199
2020
-
[17]
Niraula, Saurab Dulal, and Diwa Koirala
Nobal B. Niraula, Saurab Dulal, and Diwa Koirala. 2021. https://doi.org/10.18653/v1/2021.woah-1.7 Offensive language detection in N epali social media . In Proceedings of the 5th Workshop on Online Abuse and Harms (WOAH 2021), pages 67--75, Online. Association for Computationa...
2021 doi
-
[18]
Nobal B Niraula, Saurab Dulal, and Diwa Koirala. 2022. Linguistic taboos and euphemisms in nepali. ACM Transactions on Asian and Low-Resource Language Information Processing, 21(6):1--26
2022
-
[19]
Atul Kr Ojha. 2019. English-bhojpuri smt system: Insights from the karaka model. arXiv preprint arXiv:1905.02239
2019 arXiv
-
[20]
Kritesh Rauniyar, Sweta Poudel, Shuvam Shiwakoti, Surendrabikram Thapa, Junaid Rashid, Jungeun Kim, Muhammad Imran, and Usman Naseem. 2023. Multi-aspect annotation and analysis of nepali tweets on anti-establishment election discourse. IEEE Access
2023
-
[21]
Niloofar Safi Samghabadi, Parth Patwa, Srinivas PYKL, Prerana Mukherjee, Amitava Das, and Thamar Solorio. 2020. https://aclanthology.org/2020.trac-1.20 Aggression and misogyny detection using BERT : A multi-task approach . In Proceedings of the Second Workshop on Trolling, Agg...
2020
-
[22]
Kengatharaiyer Sarveswaran, Bal Krishna Bal, Surendrabikram Thapa, Ashwini Vaidya, and Sana Shams. 2025. A brief overview of the first workshop on challenges in processing south asian languages (chipsal). In Proceedings of the First Workshop on Challenges in Processing South A...
2025
-
[23]
Yuqing Tang, Chau Tran, Xian Li, Peng - Jen Chen, Naman Goyal, Vishrav Chaudhary, Jiatao Gu, and Angela Fan. 2020. https://arxiv.org/abs/2008.00401 Multilingual translation with extensible multilingual pretraining and finetuning . CoRR, abs/2008.00401
2020 arXiv
-
[24]
Surendrabikram Thapa, Kritesh Rauniyar, Farhan Ahmad Jafri, Surabhi Adhikari, Kengatharaiyer Sarveswaran, Bal Krishna Bal, Hariram Veeramani, and Usman Naseem. 2025. Natural language understanding of devanagari script languages: Language identification, hate speech and its tar...
2025
-
[25]
Surendrabikram Thapa, Kritesh Rauniyar, Shuvam Shiwakoti, Sweta Poudel, Usman Naseem, and Mehwish Nasim. 2023. Nehate: Large-scale annotated data shedding light on hate speech in nepali local election discourse. In ECAI 2023, pages 2346--2353. IOS Press
2023
-
[26]
Ziqing Yang et al. 2020. Mc-bert4hate: Hate speech detection using multi-channel bert for different languages and translations. In Proceedings of the 12th Language Resources and Evaluation Conference (LREC), pages 2159--2166. European Language Resources Association (ELRA)
2020
-
[27]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
-
[28]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.