On Importance of Code-Mixed Embeddings for Hate Speech Identification

Gauri Takalikar; Mihir Inamdar; Omkar Khade; Raviraj Joshi; Shruti Jagdale

REVIEW 4 major objections 5 minor 14 references

On Importance of Code-Mixed Embeddings for Hate Speech Identification

T0 review · 4 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that code-mixed Hindi-English embeddings (HingBERT and Hing-FastText) trained on L3Cube-HingCorpus outperform monolingual BERT and English FastText for hate speech detection on HASOC 2021 and an in-house HATE dataset.

desk verdict Plausible HingBERT-over-BERT result, but the manuscript hides every detail that would let you check it—empty preprocessing section, no hyperparameters, no numeric results, and a wrong BERT parameter count. read the letter →

arxiv 2411.18577 v1 pith:6SQYMNWN submitted 2024-11-27 cs.CL cs.LG

Shruti Jagdale , Omkar Khade , Gauri Takalikar , Mihir Inamdar , Raviraj Joshi This is my paper · ORCID

classification cs.CLcs.LG

keywords code-mixedtexthatespeechdetectionHingBERTHing-FastHinglishwordembeddingsHASOC

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Code-mixed Hindi-English (Hinglish) is common in Indian social media, but most NLP models are trained on monolingual English and struggle with it. This paper tries to establish that embeddings pre-trained on a large code-mixed Hindi-English corpus are better for hate speech detection than standard English-only embeddings. Using the HASOC 2021 hate speech dataset and an in-house HATE dataset, the authors report that HingBERT and Hing-FastText achieve higher F1 scores, recall, and accuracy than BERT and English FastText. If the claim holds, the practical lesson is that code-mixed pre-training, not architecture alone, is what makes hate speech models work in multilingual communities.

What carries the argument

The central object is the code-mixed embedding. HingBERT is a transformer-based language model pre-trained on the L3Cube-HingCorpus with contextual, bidirectional representations; Hing-FastText is a subword-level embedding model trained on the same corpus in Roman and Devanagari scripts. Both carry vocabulary and syntax from actual Hinglish text, which is what the paper credits for the measured hate speech detection advantage over models trained only on English.

What would settle it

Run HingBERT, Hing-FastText, BERT, and English FastText through one identical preprocessing and hyperparameter search on a held-out Hinglish hate speech test set that is verified disjoint from L3Cube-HingCorpus; if the code-mixed models no longer beat the English-only ones on F1, the paper's central claim is disproved.

Watch

Extended reading notes

Core claim

The central claim is that code-mixed representations are the decisive ingredient for hate speech identification in Hinglish: HingBERT and Hing-FastText, both trained on the L3Cube-HingCorpus (52.93 million sentences of Roman and Devanagari Hindi-English text), outperform standard BERT, multilingual BERT, and vanilla FastText on two hate speech test sets, HASOC 2021 and the authors' HATE dataset. The paper reports the code-mixed models winning on F1, recall, and accuracy, and attributes the gain to their ability to handle mixed-code context, subword information, and the real-world vocabulary of Hinglish social media.

Load-bearing premise

The central claim presupposes a fair and fully specified comparison—identical preprocessing and classifier tuning across all models, with no overlap between the L3Cube-HingCorpus pre-training data and the HASOC or HATE test sets—yet the paper does not report the preprocessing details or classifier settings.

Editorial extensions

If this is right

If correct, hate speech detection systems for Hinglish should start from code-mixed pre-trained models like HingBERT rather than English BERT.
The reported results make Hing-FastText a lightweight alternative that beats vanilla BERT, which matters for deployment where transformer compute is limited.
The findings imply that the value of a multilingual corpus is not just language coverage but alignment with the code-mixed register of the target task.
The HASOC 2021 and HATE scores serve as baselines for future work on code-mixed hate speech detection.
The advantage of HingBERT over multilingual BERT suggests that a dedicated code-mixed pre-training corpus is more useful than a broad multilingual Wikipedia-trained model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's own evidence would be more convincing if the preprocessing and classifier hyperparameters were disclosed, since Section 4.2 is empty and Section 4.4 lists no settings; a controlled replication with identical settings would test whether the reported advantage survives.
The same code-mixed pre-training recipe could transfer to other language pairs, such as Spanish-English or Arabic-French, where social media text mixes languages and monolingual models underperform.
The reported results also suggest that code-mixed embeddings may improve other Hinglish social media tasks, such as sentiment analysis, misinformation detection, or identifying in-group slurs that shift meaning across languages.
A natural next experiment is an ablation that trains HingBERT on only the Roman subset or only the Devanagari subset of L3Cube-HingCorpus to see which portion drives the hate speech gain.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Here's the short version: this paper wants to show that code-mixed HingBERT beats English BERT for Hinglish hate speech detection, and that result may well be true—but the manuscript as written doesn't let you check it. Section 4.2 (Preprocessing) is empty. Section 4.4 lists Random Forest, Linear Regression, SVM and KNN as candidate classifiers without saying which were used or with what hyperparameters. Results exist only as figures, with no numeric F1, precision, recall, or confidence intervals. The HATE dataset is never sourced. BERT-base is described as having 345 million parameters; the real number is 110 million. This is not a paper you can evaluate; it's a slide deck with an abstract.

What's good: the question is a fair one, and the comparison to HASOC 2021 is a sensible external benchmark. The Hing-FastText versus vanilla FastText comparison is a small extension beyond earlier L3Cube work, and the authors cite the key prior papers (Nayak and Joshi 2021, Farooqi et al. 2021) that already compared mBERT and Indic-BERT on the same task. So the framing is honest about where the result comes from.

The soft spot is the missing support, not the research direction. The stress-test note is right: since HingBERT and BERT use different tokenizers and hidden sizes, small choices in pooling or SVM settings could reverse the ranking. Without a fixed preprocessing and training protocol applied to all models, the reported advantage could be a tuning artifact. The authors' own lab built the Hing models, which isn't circular because the benchmark is external, but it makes full reporting more important, not less.

Proportionately: this is a reject-for-incompleteness, not a reject-for-wrongness. The claim could survive a rigorous re-run.

Recommendation: desk reject in current form. If the authors resubmit with the preprocessing details, the actual classifier and hyperparameters, and per-model numbers, it becomes a legitimate workshop paper. Not worth referee time as is.

Referee Report

4 major / 5 minor

Summary. The manuscript compares BERT-family transformers and FastText embeddings, including code-mixed Hing models developed by L3Cube, for hate speech detection on the HASOC 2021 Hindi-English dataset and a smaller in-house HATE dataset. The central claim is that HingBERT variants and Hing-FastText outperform standard English BERT and FastText models on this task. The paper describes the models, gives a brief literature survey, outlines a methodology that includes embedding generation and classification, and concludes with the comparative advantage of code-mixed models.

Significance. If the empirical claims are correct, the paper would provide useful evidence that code-mixed pre-training (L3Cube-HingCorpus) improves hate speech detection for Hindi-English text, a socially important task. The use of the external HASOC 2021 benchmark avoids an obviously circular evaluation, and the range of compared models is appropriate. However, the manuscript as submitted does not report the actual numerical results, does not describe the preprocessing or classifier configuration, and leaves the central comparison unverifiable. The significance is therefore conditional on the authors supplying the missing experimental evidence.

major comments (4)

[Section 4.2] The 'Preprocessing' subsection is empty. Since the paper's central claim is an empirical comparison, the preprocessing pipeline is load-bearing: different normalization, tokenization, handling of Devanagari script, emoji, URL, and case-folding choices can affect transformer and FastText models differently. The manuscript must state exactly what preprocessing was applied and confirm that it was identical for all compared models.
[Section 4.4 and Section 5] The classification protocol is underspecified. Section 4.4 lists Random Forest, Linear Regression, SVM, and KNN as candidate algorithms but does not state which were actually used, what hyperparameters were chosen, how train/validation/test splits were formed, or whether any hyperparameter tuning was performed. Section 5 adds that 'mostly max pooling is used' without defining when pooling differs. Without a fixed, fully specified protocol applied identically to all models, the reported Hing advantage could be a tuning artifact rather than an effect of code-mixed embeddings.
[Section 5] Results are presented only as aggregate figures ('testing scores are shown in Figures 2 and 3') with no numeric F1, precision, recall, or accuracy values for any model or dataset. This makes the abstract's claims—that HingBERT 'outperform BERT models' and that Hing-FastText 'performs better than standard English FastText and vanilla BERT models'—impossible to check or reproduce. The authors should include tables of exact numbers, with confidence intervals or standard deviations if multiple runs were performed.
[Section 4.1, Tables 1 and 2] The HATE dataset is not sourced, described, or released. The tables give category counts for only 'a portion' of the dataset, and the total size, collection method, annotation procedure, and license are absent. Without provenance and full statistics, the results cannot be reproduced, and it is unclear whether the dataset is public or whether the evaluation set overlaps with the pre-training corpus. These details are necessary to rule out data leakage and to allow independent verification.

minor comments (5)

[Section 2.1] The text states that BERT-base has 345 million parameters; BERT-base actually has about 110 million parameters, while BERT-large has about 340 million. This should be corrected together with the associated citation.
[Figures] The manuscript refers to 'Fig. 1. Flowchart' and to 'Figures 2 and 3' as comparison charts, but none of these figures appear in the text. The figures must be included, or the references removed.
[Abstract and Conclusion] The abstract and conclusion assert that Hing models outperform BERT and vanilla FastText, but no numerical evidence is given anywhere in the paper. The claims should be backed by the reported numbers or softened to what the results actually support.
[Section 2.4] The description of HingCorpus says it contains '52.93 million sentences and 1.04 billion tags'; it is unclear whether 'tags' is a typo for 'tokens', and it is not specified which subset (Roman, Devanagari, or both) was used for each Hing model in this study.
[References] Several in-text citations appear mismatched with the reference list: for example, reference [10] is described as a character-level GRU hate speech paper, but the reference list entry [10] is the Sabty et al. Arabic-English NER paper. The entire bibliography should be checked against the citations.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the Hing-model advantage is an empirical benchmark claim against the external HASOC 2021 dataset and an in-house HATE set, with no fitted prediction or self-citation chain reducing the claim to its inputs.

full rationale

The paper's central claim is an empirical comparison ('HingBERT models... outperform BERT models'; 'Hing-FastText performs better than standard English FastText and vanilla BERT models', Abstract). Nothing in the manuscript derives these comparisons from the definition of the models or from a fitted parameter; the models are pre-trained artifacts evaluated on benchmark data. The L3Cube-HingCorpus and Hing models are author-produced, but the benchmark datasets (HASOC 2021 and the Hate dataset) are external to the comparison, so the result is falsifiable and not forced by construction. The self-citation to prior L3Cube work (e.g., Ref. [1] by Nayak and Joshi) appears in the literature survey, but the paper does not invoke any uniqueness theorem or prior derivation as the basis for its empirical outcome. The manuscript does have serious reporting gaps: Section 4.2 ('Preprocessing') is empty, Section 4.4 lists candidate classifiers without stating which were used or their hyperparameters, and Section 5 reports only figures with no numeric F1/precision/recall or confidence intervals. These are missing-support and reproducibility problems that could make the claimed advantage unverifiable, but they are not circularity: the stated claim still has independent empirical content that a rigorous re-run could confirm or refute.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new mathematical objects. Its claims rest on the unverified fairness of an empirical comparison and on the completeness of the reported figures.

free parameters (3)

Classifier hyperparameters
Random Forest, SVM, KNN, and Linear Regression settings are not reported; results depend on these choices.
Pooling strategy = max pooling
Section 5 states max pooling is used, but no comparison or justification is given; pooling affects the embedding vector.
Preprocessing choices
Section 4.2 is empty, so any cleaning, normalization, or de-emoji steps are unspecified.

assumptions (3)

domain assumption L3Cube-HingCorpus pre-training data is disjoint from the HASOC and HATE test sets.
If HingCorpus contains the test tweets, the reported Hing model gains could reflect memorization. Not discussed in the paper.
domain assumption All models are compared under identical preprocessing and hyperparameter settings.
Sections 4.2 to 4.4 do not specify settings, so the fairness of the comparison is unverified.
domain assumption The figures report the complete set of runs and metrics.
Only aggregate charts are shown, with no per-model numeric tables or error bars.

how reviews work

0 comments

Cite this review

Pith. "Pith review of On Importance of Code-Mixed Embeddings for Hate Speech Identification." pith.science (2026). https://pith.science/paper/6SQYMNWN

@misc{pith2026241118577,
  author       = {Pith},
  title        = {Pith review of: On Importance of Code-Mixed Embeddings for Hate Speech Identification},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6SQYMNWN}},
  note         = {Machine review of arXiv:2411.18577}
}

read the original abstract

Code-mixing is the practice of using two or more languages in a single sentence, which often occurs in multilingual communities such as India where people commonly speak multiple languages. Classic NLP tools, trained on monolingual data, face challenges when dealing with code-mixed data. Extracting meaningful information from sentences containing multiple languages becomes difficult, particularly in tasks like hate speech detection, due to linguistic variation, cultural nuances, and data sparsity. To address this, we aim to analyze the significance of code-mixed embeddings and evaluate the performance of BERT and HingBERT models (trained on a Hindi-English corpus) in hate speech detection. Our study demonstrates that HingBERT models, benefiting from training on the extensive Hindi-English dataset L3Cube-HingCorpus, outperform BERT models when tested on hate speech text datasets. We also found that code-mixed Hing-FastText performs better than standard English FastText and vanilla BERT models.

Figures

Figures reproduced from arXiv: 2411.18577 by the authors.

**Figure 1.** Flowchart We are using a transformer model that can handle two languages, which brings a significant increase in efficiency and performance compared to previous methods. The models we are using include BERT, Multilingual BERT, DistilBERT, RoBERTa, HingBERT, Hing-mBERT, Hing-RoBERTa, HingGPT, HingBERT-LID and HingFT. The process involves encoding the input text sequence using the hidden state of the transformer arc… view at source ↗

**Figure 2.** Comparison Chart for HASOC dataset [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

**Figure 3.** Comparison Chart for HATE dataset 6 Conclusion This paper discusses the use of specialized software to detect hate speech in multilingual contexts with a particular focus on Hindi-English (Hinglish) texts prevalent in India. This study demonstrates the superior performance of HingBERT trained on the L3Cube-HingCorpus database in detecting hate speech due to its ability to handle mixed code context. Additionally, th… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

14 extracted references · 11 canonical work pages

[1]

Contextual Hate Speech Detection in Code Mixed Text using Transformer Based Approaches

Nayak, Ravindra and Raviraj Joshi. “Contextual Hate Speech Detection in Code Mixed Text using Transformer Based Approaches.” Fire (2021)

work page 2021
[2]

Leveraging Transformers for Hate Speech Detection in Conversational Code-Mixed Tweets

Farooqi, Zaki Mustafa et al. “Leveraging Transformers for Hate Speech Detection in Conversational Code-Mixed Tweets.” Fire (2021)

work page 2021
[3]

Word Level Language Identification in Code-Mixed Data using Word Embedding Methods for Indian 8 Shruti et al. Languages,

I. Chaitanya, I. Madapakula, S. K. Gupta and S. Thara, "Word Level Language Identification in Code-Mixed Data using Word Embedding Methods for Indian 8 Shruti et al. Languages," 2018 International Conference on Advances in Computing, Commu- nications and Informatics (ICACCI), Bangalore, India, 2018, pp. 1137-1141, doi: 10.1109/ICACCI.2018.8554501

work page arXiv 2018
[4]

A Comparison Study of Word Em- bedding for Detecting Named Entities of Code-Mixed Data in Indian Lan- guage,

L. Sravani, A. S. Reddy and S. Thara, "A Comparison Study of Word Em- bedding for Detecting Named Entities of Code-Mixed Data in Indian Lan- guage," 2018 International Conference on Advances in Computing, Communi- cations and Informatics (ICACCI), Bangalore, India, 2018, pp. 2375-2381, doi: 10.1109/ICACCI.2018.8554918

work page arXiv 2018
[5]

Comparison of Pretrained Embeddings to Identify Hate Speech in Indian Code-Mixed Text,

S. Banerjee, B. Raja Chakravarthi and J. P. McCrae, "Comparison of Pretrained Embeddings to Identify Hate Speech in Indian Code-Mixed Text," 2020 2nd Inter- national Conference on Advances in Computing, Communication Control and Net- working (ICACCCN), Greater Noida, India, 2020, pp. 21-25, doi: 10.1109/ICAC- CCN51052.2020.9362731

work page arXiv 2020
[6]

Arra’Di Nur Rizal and Sara Stymne. 2020. Evaluating Word Embeddings for In- donesian–English Code-Mixed Text Based on Synthetic Data. In Proceedings of the The 4th Workshop on Computational Approaches to Code Switching, pages 26–35, Marseille, France. European Language Resources Association

work page 2020
[7]

Named Entity Recognition on Arabic-English Code-Mixed Data,

C. Sabty, M. Elmahdy, and S. Abdennadher, "Named Entity Recognition on Arabic-English Code-Mixed Data," in 13th International Conference on Seman- tic Computing (ICSC), Newport Beach, CA, USA, 2019, pp. 93–97

work page 2019
[8]

Pre-trained Word Embeddings for Malayalam Language: A Review,

K. Reji Rahmath, P. C. R. Raj and P. C. Rafeeque, "Pre-trained Word Embeddings for Malayalam Language: A Review," 2021 International Conference on Artificial Intelligence and Smart Systems (ICAIS), Coimbatore, India, 2021, pp. 568-572, doi: 10.1109/ICAIS50930.2021.9396042

arXiv 2021

Show all 14 references

[9]

Adithya Pratapa, Monojit Choudhury, and Sunayana Sitaram. 2018. Word Embed- dings for Code-Mixed Language Processing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3067–3072, Brussels, Belgium. Association for Computational Li...

2018
[10]

Named Entity Recognition on Arabic-English Code-Mixed Data,

C. Sabty, M. Elmahdy and S. Abdennadher, "Named Entity Recognition on Arabic-English Code-Mixed Data," 2019 IEEE 13th International Conference on Semantic Computing (ICSC), Newport Beach, CA, USA, 2019, pp. 93-97, doi: 10.1109/ICOSC.2019.8665500

2019
[11]

Sentiment Analysis of Persian-English Code-mixed Texts,

N. Sabri, A. Edalat and B. Bahrak, "Sentiment Analysis of Persian-English Code-mixed Texts," 2021 26th International Computer Conference, Computer Society of Iran (CSICC), Tehran, Iran, 2021, pp. 1-4, doi: 10.1109/CS- ICC52343.2021.9420605

2021
[12]

Hindi-English Code Mixed Hate Speech Detection using Character Level Embeddings,

Rahul, V. Gupta, V. Sehra and Y. R. Vardhan, "Hindi-English Code Mixed Hate Speech Detection using Character Level Embeddings," 2021 5th International Con- ference on Computing Methodologies and Communication (ICCMC), Erode, India, 2021, pp. 1112-1118, doi: 10.1109/ICCMC51019....

2021
[13]

In Proceedings ofthe4thWorkshoponRepresentationLearningforNLP(RepL4NLP-2019),pages 181–186, Florence, Italy

GentaIndraWinata,ZhaojiangLin,andPascaleFung.2019.LearningMultilingual Meta-Embeddings for Code-Switching Named Entity Recognition. In Proceedings ofthe4thWorkshoponRepresentationLearningforNLP(RepL4NLP-2019),pages 181–186, Florence, Italy. Association for Computational Linguistics

2019
[14]

Pranaydeep Singh and Els Lefever. 2020. Sentiment Analysis for Hinglish Code- mixed Tweets by means of Cross-lingual Word Embeddings. In Proceedings of the The 4th Workshop on Computational Approaches to Code Switching, pages 45–51, Marseille, France. European Language Resourc...

2020

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Contextual Hate Speech Detection in Code Mixed Text using Transformer Based Approaches

Nayak, Ravindra and Raviraj Joshi. “Contextual Hate Speech Detection in Code Mixed Text using Transformer Based Approaches.” Fire (2021)

work page 2021

[2] [2]

Leveraging Transformers for Hate Speech Detection in Conversational Code-Mixed Tweets

Farooqi, Zaki Mustafa et al. “Leveraging Transformers for Hate Speech Detection in Conversational Code-Mixed Tweets.” Fire (2021)

work page 2021

[3] [3]

Word Level Language Identification in Code-Mixed Data using Word Embedding Methods for Indian 8 Shruti et al. Languages,

I. Chaitanya, I. Madapakula, S. K. Gupta and S. Thara, "Word Level Language Identification in Code-Mixed Data using Word Embedding Methods for Indian 8 Shruti et al. Languages," 2018 International Conference on Advances in Computing, Commu- nications and Informatics (ICACCI), Bangalore, India, 2018, pp. 1137-1141, doi: 10.1109/ICACCI.2018.8554501

work page arXiv 2018

[4] [4]

A Comparison Study of Word Em- bedding for Detecting Named Entities of Code-Mixed Data in Indian Lan- guage,

L. Sravani, A. S. Reddy and S. Thara, "A Comparison Study of Word Em- bedding for Detecting Named Entities of Code-Mixed Data in Indian Lan- guage," 2018 International Conference on Advances in Computing, Communi- cations and Informatics (ICACCI), Bangalore, India, 2018, pp. 2375-2381, doi: 10.1109/ICACCI.2018.8554918

work page arXiv 2018

[5] [5]

Comparison of Pretrained Embeddings to Identify Hate Speech in Indian Code-Mixed Text,

S. Banerjee, B. Raja Chakravarthi and J. P. McCrae, "Comparison of Pretrained Embeddings to Identify Hate Speech in Indian Code-Mixed Text," 2020 2nd Inter- national Conference on Advances in Computing, Communication Control and Net- working (ICACCCN), Greater Noida, India, 2020, pp. 21-25, doi: 10.1109/ICAC- CCN51052.2020.9362731

work page arXiv 2020

[6] [6]

Arra’Di Nur Rizal and Sara Stymne. 2020. Evaluating Word Embeddings for In- donesian–English Code-Mixed Text Based on Synthetic Data. In Proceedings of the The 4th Workshop on Computational Approaches to Code Switching, pages 26–35, Marseille, France. European Language Resources Association

work page 2020

[7] [7]

Named Entity Recognition on Arabic-English Code-Mixed Data,

C. Sabty, M. Elmahdy, and S. Abdennadher, "Named Entity Recognition on Arabic-English Code-Mixed Data," in 13th International Conference on Seman- tic Computing (ICSC), Newport Beach, CA, USA, 2019, pp. 93–97

work page 2019

[8] [8]

Pre-trained Word Embeddings for Malayalam Language: A Review,

K. Reji Rahmath, P. C. R. Raj and P. C. Rafeeque, "Pre-trained Word Embeddings for Malayalam Language: A Review," 2021 International Conference on Artificial Intelligence and Smart Systems (ICAIS), Coimbatore, India, 2021, pp. 568-572, doi: 10.1109/ICAIS50930.2021.9396042

arXiv 2021