Pith. sign in

REVIEW 2 major objections 6 minor 22 references

IITR-CIOL@NLU of Devanagari Script Languages 2025: Multilingual Hate Speech Detection and Target Identification in Devanagari-Scripted Languages

T0 review · 2 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Fine-tuning the ia-multilingual-transliterated-roberta base as a single classifier detects hate speech across five Devanagari-scripted languages with 88.40% test accuracy, while target identification reaches 66.11%.

desk verdict A shared-task write-up with plausible external test numbers but an internal model description that cannot produce the three-way Subtask C results as written. read the letter →

arxiv 2412.17947 v2 pith:OBX5M3MG submitted 2024-12-23 cs.CL

Siddhant Gupta , Siddh Singhal , Azmine Toushik Wasi This is my paper · ORCID
classification cs.CL
keywords hatespeechdetectionDevanagariscriptmultilingualtransformertargetidentificationlow-resourcelanguagesfine-tuningSouthAsiantransliteration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports a direct system for two shared tasks on Devanagari-scripted South Asian languages: detecting whether a tweet contains hate speech, and identifying the target of that hate as an individual, organization, or community. Its claim is that one multilingual transformer, fine-tuned as MultilingualRobertaClass without any language-specific components, can handle all five languages, Hindi, Marathi, Nepali, Bhojpuri, and Sanskrit, at once. On the test set the model reaches 88.40% accuracy for hate speech detection and 66.11% for target identification, with the precision-recall numbers weaker than accuracy implies. A sympathetic reader would take the paper as evidence that shared-representation fine-tuning is enough for usable hate speech detection in these under-resourced languages, and that target identification remains the harder open problem.

What carries the argument

The load-bearing object is the MultilingualRobertaClass model, built on the pretrained multilingual transformer ia-multilingual-transliterated-roberta, which supplies contextualized embeddings for all five languages from shared syntactic and semantic structure. The transformer's CLS token is passed through a linear pre-classifier that keeps the 768-dimensional representation, a ReLU activation, and a dropout layer with rate 0.3, then a final linear layer with sigmoid activation produces the classification probability. This machinery is what lets a single model see Devanagari and transliterated text from different languages without per-language adapters, and the fine-tuning procedure, AdamW with learning rate $2\times10^{-5}$, batch size 16, and two to five epochs, is what adapts the pretrained knowledge to the hate speech task.

What would settle it

Split the shared-task test set by language and compute accuracy and F1 for Hindi, Marathi, Nepali, Bhojpuri, and Sanskrit separately; if a low-resource language lands near chance, around 50% for Subtask B or 33% for Subtask C, while Hindi dominates the aggregate, the multilingual claim is not supported for that language.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that a deep neural network formed by wrapping a lightweight classifier head around the pretrained multilingual transliterated RoBERTa model can classify Devanagari-scripted text across five languages using one shared set of weights. For Subtask B, hate speech detection, the test set gives accuracy 0.8840, precision 0.7106, recall 0.6547, and F1 0.6762; for Subtask C, target identification, accuracy is 0.6611, precision 0.5910, recall 0.5839, and F1 0.5816. The authors read the gap as showing that hate speech detection is already tractable with current multilingual pretraining, while identifying the target of hate speech needs more data or more specialized model structure. Ablation experiments on sequence length, learning rate, and batch size show sequence length to be the most consequential hyperparameter for both subtasks.

Load-bearing premise

The system works only insofar as the pretrained multilingual model already encodes all five languages well enough, and because there is no per-language adaptation and no per-language evaluation, poor representation of any one language would silently lower the pooled numbers.

Editorial extensions

If this is right

  • A single shared classifier is enough to detect hate speech across the five Devanagari-scripted languages in the benchmark.
  • Target identification is substantially harder; the paper attributes the gap to task complexity and the need for deeper contextual understanding.
  • Sequence length is the most influential hyperparameter in both subtasks, so keeping the 256-token budget matters more than tuning learning rate or batch size.
  • Current multilingual pretrained representations are sufficient for binary hate classification but are not yet sufficient for fine-grained target categorization.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper reports only pooled scores, a natural test is per-language evaluation; if Bhojpuri or Sanskrit accuracy is near chance, the headline numbers overstate coverage for the lowest-resource languages.
  • The gap between accuracy 0.8840 and F1 0.6762 in Subtask B suggests a class-imbalanced test set, and a model tuned for F1 rather than accuracy would likely present a different trade-off.
  • The same architecture could be applied to other Brahmic scripts to test whether the transliterated multilingual base transfers beyond Devanagari.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. The paper describes the IITR-CIOL submission to the CHiPSAL 2025 shared task on natural language understanding of Devanagari-scripted languages. The authors fine-tune the multilingual transformer ia-multilingual-transliterated-roberta with a lightweight classification head (pre-classifier, ReLU, dropout, classifier) and evaluate it on Subtask B (binary hate-speech detection) and Subtask C (three-way target identification: individual, organization, community). They report validation results and hyperparameter ablations, and test-set accuracy of 0.8840 for Subtask B and 0.6611 for Subtask C. The paper includes a Limitations section acknowledging potential under-representation of low-resource languages in the pretrained model.

Significance. The reported test numbers come from the official external shared-task test set, which is a genuine strength and avoids circularity. If the architectural inconsistency for Subtask C is resolved, the paper would provide a useful multilingual baseline for hate-speech detection and target identification in Devanagari scripts. Its value is primarily as a system description; it does not claim methodological novelty. The explicit hyperparameter list and the candid Limitations section are positive features. However, the internal inconsistency between the binary-head description and the three-class Subtask C results, together with impossible micro-recall entries in the ablation tables, currently prevents acceptance of the reported results at face value.

major comments (2)
  1. [§2.2, §2.3, Table 3] Sections 2.2 and 2.3 specify a classification head that maps the 768-dimensional representation to a single scalar output with a sigmoid activation, and refer to binary classification with cross-entropy loss. Subtask C, however, is defined in Section 2.1 as a three-way classification into individual, organization, or community targets. A single sigmoid output cannot represent a three-class categorical decision, so as written this architecture cannot be the system that produced the Subtask C test results in Table 3 (accuracy 0.6611, F1 0.5816). Please specify the actual output head used for Subtask C, for example three logits with softmax, or clarify whether a separate multi-class model was trained. Without this clarification, the Subtask C numbers are unattributable to the described model.
  2. [Tables 1 and 2] The micro-recall columns in the ablation tables are internally inconsistent with the accuracy and micro-precision columns. For single-label classification, micro-averaged precision, micro-averaged recall, and accuracy are all equal to the overall fraction of correct predictions. For example, Table 1 reports accuracy 0.8050 and micro precision 0.8050 but micro recall 0.8100 for the sequence-length-128 variant; Table 2 reports accuracy 0.7250 and micro precision 0.7250 but micro recall 0.7300 for the learning-rate variant. These values cannot all be correct. Please recompute or correct the tables and state the micro-averaging formula used.
minor comments (6)
  1. [Limitations] The Limitations section acknowledges that the pretrained model may not fully capture Devanagari-scripted language nuances, but the paper provides no per-language breakdown for Hindi, Marathi, Nepali, Bhojpuri, and Sanskrit. If the shared task evaluation does not provide per-language labels, the authors should state this explicitly; otherwise reporting per-language accuracy or error analysis would substantiate the multilingual claim.
  2. [§2.3] The sentence 'a pre-trained transformer model (such as BERT (Devlin et al., 2019))' is misleading because the actual base model is ia-multilingual-transliterated-roberta; please state the base model name directly in the implementation details.
  3. [§2.3] The phrase 'gradient clipping is likely employed' is too hedged for a system description; please either specify the exact clipping norm and strategy or remove the claim.
  4. [Tables 1 and 2] The weighting scheme for 'weighted precision,' 'weighted recall,' and 'weighted F1' is not defined; please specify whether the weights are class frequencies, support sizes, or something else.
  5. [§3.2] The text contains a typo: 'It shows that,.' should read 'It shows that'; also '10−5causes' needs a space, and the abstract and Section 5 state that the classifier head is for binary classification even though Subtask C is three-class.
  6. [General] The paper reports single-run results without variance estimates or significance testing; given the small differences among ablation variants (e.g., 0.8180 vs 0.8150 in Table 1), it is unclear which differences are meaningful. Please state how many seeds or runs were averaged, or report that each configuration was run once.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: reported scores are external shared-task test results; the only self-citation is non-load-bearing. Internally inconsistent head/metric descriptions are correctness risks, not circular derivations.

full rationale

The central claim is the test-set performance on the CHiPSAL 2025 shared task (Section 4, Table 3). These numbers are evaluated against an externally provided held-out test set, not fitted to any target, so the main result is self-contained and externally falsifiable. The self-citation (Wasi, 2024) appears only as general related work on hate speech and is not load-bearing for the architecture or the reported scores. The pretrained model is cited to IBM/Dhamecha et al., not to the present authors, and the fine-tuning procedure is standard, so no result is imported from a self-citation chain. The Limitations section itself acknowledges the dependence on the pretrained multilingual model, which is an honest limitation rather than a circular step. Note two non-circular reporting concerns: Section 2.2 describes a single-sigmoid binary classification head while Subtask C (Section 2.1) is a three-way target identification task, and Tables 1-2 list micro-recall values different from accuracy/micro-precision, which is inconsistent with standard micro-averaging; these are reproducibility and consistency issues, not reductions of the claimed result to its inputs.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities or mechanisms. The model is a standard pretrained transformer with a linear classifier head. The main load-bearing, hand-chosen components are the hyperparameters listed above. The model weights themselves are learned from the training data and are not free parameters in the derivation sense.

free parameters (5)
  • learning rate = 2e-5
    Chosen by hand; the reported test accuracy depends on this value.
  • batch size = 16 train, 64 eval
    Chosen by hand; no batch size search is reported.
  • max sequence length = 256 tokens
    Ablation at 128 lowers validation accuracy, so this choice affects results.
  • dropout rate = 0.3
    Chosen by hand for the classifier head.
  • number of epochs = 2-5
    Stated as a range; the exact epoch count used for test predictions is not documented.
assumptions (3)
  • domain assumption The shared task test labels are correct and representative of the target languages and domains.
    The reported accuracy assumes the test set is ground truth.
  • domain assumption ia-multilingual-transliterated-roberta provides useful representations for all five Devanagari-scripted languages.
    The system relies entirely on this pretrained model; the Limitations section itself questions this.
  • standard math Standard fine-tuning behavior: the optimizer converges and the evaluation protocol is unbiased.
    Background assumption for any fine-tuning paper; no formal proof is given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of IITR-CIOL@NLU of Devanagari Script Languages 2025: Multilingual Hate Speech Detection and Target Identification in Devanagari-Scripted Languages." pith.science (2026). https://pith.science/paper/OBX5M3MG

@misc{pith2026241217947,
  author       = {Pith},
  title        = {Pith review of: IITR-CIOL@NLU of Devanagari Script Languages 2025: Multilingual Hate Speech Detection and Target Identification in Devanagari-Scripted Languages},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OBX5M3MG}},
  note         = {Machine review of arXiv:2412.17947}
}
read the original abstract

This work focuses on two subtasks related to hate speech detection and target identification in Devanagari-scripted languages, specifically Hindi, Marathi, Nepali, Bhojpuri, and Sanskrit. Subtask B involves detecting hate speech in online text, while Subtask C requires identifying the specific targets of hate speech, such as individuals, organizations, or communities. We propose the MultilingualRobertaClass model, a deep neural network built on the pretrained multilingual transformer model ia-multilingual-transliterated-roberta, optimized for classification tasks in multilingual and transliterated contexts. The model leverages contextualized embeddings to handle linguistic diversity, with a classifier head for binary classification. We received 88.40% accuracy in Subtask B and 66.11% accuracy in Subtask C, in the test set.

Figures

Figures reproduced from arXiv: 2412.17947 by the authors.

Figure 1
Figure 1. Model architecture, containing tokenizer, pre [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 12 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Nikolaevna Alexeeva, Aleksandr Maximovich Ryabchikov, Calambur Sivaramamurti, and Yury Konstantinovich Yefremov

    N. Nikolaevna Alexeeva, Aleksandr Maximovich Ryabchikov, Calambur Sivaramamurti, and Yury Konstantinovich Yefremov. 2024. https://www.britannica.com/place/South-Asia South asia . Encyclopedia Britannica, November 6, 2024

    work page 2024

  4. [4]

    Rahul Aralikatte, Miryam De Lhoneux, Anoop Kunchukuttan, and Anders S gaard. 2021. Itihasa: A large-scale corpus for sanskrit to english translation. In Proceedings of the 8th Workshop on Asian Translation (WAT2021), pages 191--197

    work page 2021

  5. [5]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    doi:10.18653/v1/n19-1423 2019

  6. [6]

    Tejas Dhamecha, Rudra Murthy, Samarth Bharadwaj, Karthik Sankaranarayanan, and Pushpak Bhattacharyya. 2021. https://doi.org/10.18653/v1/2021.emnlp-main.675 Role of L anguage R elatedness in M ultilingual F ine-tuning of L anguage M odels: A C ase S tudy in I ndo- A ryan L anguages . In Proceedings of the 2021 Conference on Empirical Methods in Natural Lan...

    doi:10.18653/v1/2021.emnlp-main.675 2021

  7. [7]

    Farhan Ahmad Jafri, Kritesh Rauniyar, Surendrabikram Thapa, Mohammad Aman Siddiqui, Matloob Khushi, and Usman Naseem. 2024. Chunav: Analyzing hindi hate speech and targeted groups in indian election discourse. ACM Transactions on Asian and Low-Resource Language Information Processing

    work page 2024

  8. [8]

    Farhan Ahmad Jafri, Mohammad Aman Siddiqui, Surendrabikram Thapa, Kritesh Rauniyar, Usman Naseem, and Imran Razzak. 2023. Uncovering political hate speech during indian election campaign: A new low-resource dataset and baselines

    work page 2023

Show all 22 references
  1. [9]

    Raviraj Joshi. 2022. https://doi.org/10.13140/RG.2.2.14606.84809 L3cube-hindbert and devbert: Pre-trained bert transformer models for devanagari based hindi and marathi languages

    2022

  2. [10]

    Atharva Kulkarni, Meet Mandhane, Manali Likhitkar, Gayatri Kshirsagar, and Raviraj Joshi. 2021. L3cubemahasent: A marathi tweet-based sentiment analysis dataset. In Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Ana...

    2021

  3. [11]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692

    2019 arXiv

  4. [12]

    Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Zhiqiang Hu, Chenhui Shen, Yew Ken Chia, Xingxuan Li, Jianyu Wang, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang, Chaoqun Liu, Hang Zhang, and Lidong Bing. 2024. https://doi.org/10.18653/v1/2024.acl-demos....

    2024 doi

  5. [13]

    Atul Kr Ojha. 2019. English-bhojpuri smt system: Insights from the karaka model. arXiv preprint arXiv:1905.02239

    2019 arXiv

  6. [14]

    The pandas development team. 2020. https://doi.org/10.5281/zenodo.3509134 pandas-dev/pandas: Pandas

    2020 doi

  7. [15]

    Anil Singh Parihar, Surendrabikram Thapa, and Sushruti Mishra. 2021. Hate speech detection using natural language processing: Applications and challenges. In 2021 5th International Conference on Trends in Electronics and Informatics (ICOEI), pages 1302--1308. IEEE

    2021

  8. [16]

    Gita Rajan and Vinay Lal. 2007. https://doi.org/10.1080/14746680701208596 South asian popular culture: Beyond and beneath the habitual . South Asian Popular Culture, 5(1):1–10

    2007 doi

  9. [17]

    Kritesh Rauniyar, Sweta Poudel, Shuvam Shiwakoti, Surendrabikram Thapa, Junaid Rashid, Jungeun Kim, Muhammad Imran, and Usman Naseem. 2023. Multi-aspect annotation and analysis of nepali tweets on anti-establishment election discourse. IEEE Access

    2023

  10. [18]

    Kengatharaiyer Sarveswaran, Bal Krishna Bal, Surendrabikram Thapa, Ashwini Vaidya, and Sana Shams. 2025. A brief overview of the first workshop on challenges in processing south asian languages (chipsal). In Proceedings of the First Workshop on Challenges in Processing South A...

    2025

  11. [19]

    Surendrabikram Thapa, Kritesh Rauniyar, Farhan Ahmad Jafri, Surabhi Adhikari, Kengatharaiyer Sarveswaran, Bal Krishna Bal, Hariram Veeramani, and Usman Naseem. 2025. Natural language understanding of devanagari script languages: Language identification, hate speech and its tar...

    2025

  12. [20]

    Surendrabikram Thapa, Kritesh Rauniyar, Shuvam Shiwakoti, Sweta Poudel, Usman Naseem, and Mehwish Nasim. 2023. Nehate: Large-scale annotated data shedding light on hate speech in nepali local election discourse. In ECAI 2023, pages 2346--2353. IOS Press

    2023

  13. [21]

    Azmine Toushik Wasi. 2024. https://doi.org/10.18653/v1/2024.nlp4pi-1.23 Explainable identification of hate speech towards islam using graph neural networks . In Proceedings of the Third Workshop on NLP for Positive Impact, pages 250--257, Miami, Florida, USA. Association for C...

    2024 doi

  14. [22]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mari...

    2020 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.