REVIEW 4 major objections 5 minor 33 references
Automated Classification of Cybercrime Complaints using Transformer-based Language Models for Hinglish Texts
T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read HingRoBERTa, a Hinglish-adapted transformer, reaches 74.41% accuracy and 71.49% F1 on real-world cybercrime complaints, and the paper claims its pipeline is ready for deployment.
desk verdict Useful applied paper with a real dataset and a sensible privacy-aware pipeline, but the headline metrics are computed on a pruned label set, so the deployment claim needs scrutiny. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing component is HingRoBERTa, a Hinglish-adapted RoBERTa model that produces contextual embeddings for code-mixed complaint text and feeds a linear classifier over 14 standardized crime-type labels. Around this model sits the augmentation machinery: LLaMA 3.1-7b generates paraphrases of minority-class complaints after entity replacement, and a similarity filter with a 97% threshold discards sentences that drift from the original meaning; this takes the dataset from 85,875 to 109,294 samples while rebalancing rare categories such as ransomware and cyber trafficking. Privacy preprocessing—regex-based plus statistical entity recognition that replaces names, phones, emails, addresses, and monetary values with placeholder tokens—is applied before both augmentation and training, so the model learns from semantic patterns rather than memorizing individuals.
What would settle it
Manually relabel a random sample of the augmented complaints and retrain HingRoBERTa with corrected labels; if many augmented samples carry a different crime type than their source, and minority-class F1 drops under corrected training, the reported augmentation benefit is partly an artifact of label drift.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that HingRoBERTa—a RoBERTa variant pretrained on code-mixed Hindi-English text—is the best classifier for cybercrime complaints among the models tested, achieving 74.41% accuracy and 71.49% F1-score on a highly imbalanced real-world dataset of Hinglish complaints spanning 14 standardized crime types. The paper argues this result validates the importance of language adaptation, since both Hinglish-adapted models (HingBERT and HingRoBERTa) beat their generic counterparts (BERT and RoBERTa). It also claims that the GenAI-based augmentation pipeline materially improves minority-class performance: a controlled experiment without augmentation showed weaker results, especially on underrepresented categories. The paper's conclusion is that the full pipeline of language adaptation, generative rebalancing, and anonymization yields a privacy-conscious classifier that is deployable in operational settings such as national reporting portals.
Load-bearing premise
The load-bearing premise is that the LLM-generated paraphrases preserve the original crime label—the similarity filter checks wording closeness, not label correctness—so if mislabeled paraphrases enter training, the reported gains on rare crime types could be inflated.
Editorial extensions
If this is right
- For Indian cybercrime complaint triage, Hinglish-adapted transformers should be preferred over generic English-only models, because HingRoBERTa outperformed BERT and RoBERTa on the same data.
- Law-enforcement agencies can scale triage by fine-tuning this pipeline on their own complaint streams, since the preprocessing, augmentation, and deployment components are modular.
- Minority crime categories such as ransomware and cyber trafficking can be reinforced without manual data collection, through LLM paraphrase augmentation of scarce examples.
- Anonymization can be built into the training pipeline without sacrificing accuracy, which matters for compliance with data-protection rules.
- The model's performance provides a baseline for future work on code-mixed complaint classification, including hierarchical or finer-grained crime labels.
Reading between the lines
- A testable consequence of the augmentation design is that label drift could inflate minority-class F1: the 97% similarity filter checks wording closeness, not crime-category agreement, so a manual audit of augmented samples would settle whether the reported gains are real.
- Because the paper reports only aggregate metrics, per-class precision and recall on a human-verified test set would show whether the headline accuracy is driven mainly by the dominant financial-fraud class.
- The same pipeline should transfer to other code-mixed Indian languages (Banglish, Tanglish) by swapping HingRoBERTa for a matching language-adapted transformer and rerunning the augmentation and anonymization steps.
- An ablation against conventional oversampling with matched class counts would isolate whether the gains come from semantic diversity in paraphrases or simply from rebalancing.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents an automated pipeline for classifying Hinglish cybercrime complaints, combining privacy-aware preprocessing, LLaMA-based data augmentation for class imbalance, and fine-tuned transformer models (BERT, RoBERTa, HingBERT, HingRoBERTa). Using a real-world dataset from the I4C CyberGuard AI Hackathon, the authors report that HingRoBERTa achieves the highest accuracy (74.41%) and F1-score (71.49%). A Django-based deployment is described as ready for real-world use on platforms like the National Cyber Crime Reporting Portal.
Significance. If the reported results are reliable, this work addresses a genuine operational need: automated triage of code-mixed (Hinglish) cybercrime complaints while respecting privacy constraints. The paper also contributes an augmented dataset (planned release) and a practical deployment framework. The comparison of generic vs. Hinglish-adapted transformers on a private, real-world dataset is valuable for the NLP-for-security community. However, the evaluation protocol has several support gaps that must be addressed before the central deployment claim can be accepted.
major comments (4)
- [III-C.1 and Table VI] The reported accuracy (74.41%) and F1-score (71.49%) are computed on a test set from which entire categories are excluded: classes absent from the training set (e.g., Crime Against Women & Children) and classes with fewer than 2 training samples (e.g., Report Unlawful Content) are removed before evaluation. A deployed system would encounter complaints from these categories, and because the model cannot emit those labels, such failures are never reflected in the headline metrics. The paper should either evaluate on the full test set (including all original 14 categories) or explicitly report the number of excluded samples and provide per-class results for the excluded categories (e.g., as an 'other' class or with a not-classified rate). Without this, the claim of 'ready for real-world deployment' in Section V-D and the abstract is not supported.
- [V-C] Section V-C states that a controlled experiment was conducted by training HingRoBERTa without augmentation and that the resulting model showed weaker performance, 'especially on minority classes,' but no numerical results are reported. This is a load-bearing claim because the paper attributes a significant part of the improvement to augmentation. The authors should provide a table comparing HingRoBERTa with and without augmentation, including accuracy, F1, and per-class recall for minority categories, so readers can quantitatively assess the contribution of augmentation.
- [III-D] The augmentation pipeline uses LLaMA 3.1-7b to paraphrase original complaints and retains only samples with a BERTScore similarity above 97%. However, the threshold filters for semantic similarity to the source text, not for preservation of the original crime category label. There is no manual or automated verification that the generated paraphrases indeed belong to the same class. Since augmentation is applied per-class to balance the distribution, mislabeled augmented samples could inflate minority-class metrics. The authors should add a label-verification step (e.g., human evaluation on a sample, or a secondary classifier's label agreement) and report the results.
- [V-B and Table VI] Each model is evaluated on a single run, with no confidence intervals, error bars, or significance tests. The reported differences are small (e.g., F1 71.02 for HingBERT vs. 71.49 for HingRoBERTa), and without variance estimates it is unclear whether the observed ordering is reliable. At minimum, the authors should perform multiple runs (e.g., 3–5 seeds) and report mean and standard deviation, and ideally a paired significance test (e.g., McNemar's test) for the top-performing models.
minor comments (5)
- [III-B] The dataset size is written as '85, 8754' in the text; this appears to be a typo for '85,875'.
- [III-D] The similarity metric is referred to as 'BERTF1-score' but the described method (cosine similarity over BERT embeddings) is BERTScore; please use the standard terminology consistently.
- [Table III] The label standardization dictionary should state explicitly whether the same mapping is applied to both training and test labels. If test labels are standardized while the original task used the finer-grained labels, the reported metrics reflect a modified task rather than the original complaint categories.
- [References] References [6] and [13] appear to refer to the same work (TTPXHunter) but with different titles and venues; please unify or disambiguate.
- [II and Table I] The related work comparison in Table I omits dataset size for Prabhu et al. ('−') and does not list the number of classes; consider adding these details for a fairer comparison.
Circularity Check
Empirical benchmark with no derivation-level circularity; reported metrics are computed on held-out test data and are not reconstructed from fitted parameters. Minor self-citations exist but are not load-bearing.
full rationale
The paper's central claim—that HingRoBERTa achieves 74.41% accuracy and 71.49% F1 on Hinglish cybercrime complaints—rests on a standard fine-tuning and evaluation pipeline. Models are trained on an augmented training set and scored on a separate test set (Sections III-B, IV, V-B); no equation reconstructs the reported metric from a fitted parameter or from the model's own training objective. The GenAI augmentation step (Section III-D) is described self-containedly: LLaMA 3.1-7b paraphrases complaints, a BERTScore-style similarity filter keeps samples above a 97% threshold, and the expanded training set is used only for training. The label standardization and entity anonymization are preprocessing transformations, not predictions. The paper cites prior work by the same authors, e.g., [6] and [13], to motivate contextual augmentation and TTP extraction, but those citations are not the evidence for the benchmark numbers, which are computed from the I4C dataset and compared against independently defined baselines. The self-citations are therefore not load-bearing. Two evaluation-validity issues should be separated from circularity: Section III-C.1 excludes categories absent from training or with fewer than two training samples from evaluation, and Section V-C mentions a no-augmentation control without reporting its numerical results. These concerns affect how representative the 74.41%/71.49% figures are for real-world deployment, but they do not make the reported numbers equivalent to the inputs by construction. No circular step satisfying the quote-and-reduction standard was found.
Assumptions & free parameters
free parameters (4)
- BERTScore similarity threshold =
0.97
- Per-class augmentation counts =
Various (e.g., +4806 for Sexually Obscene Content, +6382 for Hacking/Damage)
- Fine-tuning hyperparameters =
Not fully specified (learning rate 1e-5 to 3e-5, batch size 8 to 32, sequence length 128 or 256)
- Label standardization mapping =
Table III mapping
assumptions (5)
- domain assumption BERTScore is a reliable proxy for semantic similarity and label preservation.
- domain assumption LLaMA 3.1-7b paraphrases of a complaint belong to the same crime category as the original.
- domain assumption The I4C hackathon dataset labels are accurate and the simplified labels in Table III are consistent.
- domain assumption The validation split is representative of the test distribution.
- domain assumption spaCy and regex entity replacement removes PII without changing the crime-related semantics.
Cite this review
Pith. "Pith review of Automated Classification of Cybercrime Complaints using Transformer-based Language Models for Hinglish Texts." pith.science (2026). https://pith.science/paper/GTYZJYUX
@misc{pith2026241216614,
author = {Pith},
title = {Pith review of: Automated Classification of Cybercrime Complaints using Transformer-based Language Models for Hinglish Texts},
year = {2026},
howpublished = {\url{https://pith.science/paper/GTYZJYUX}},
note = {Machine review of arXiv:2412.16614}
}
read the original abstract
The rise in cybercrime and the complexity of multilingual and code-mixed complaints present significant challenges for law enforcement and cybersecurity agencies. These organizations need automated, scalable methods to identify crime types, enabling efficient processing and prioritization of large complaint volumes. Manual triaging is inefficient, and traditional machine learning methods fail to capture the semantic and contextual nuances of textual cybercrime complaints. Moreover, the lack of publicly available datasets and privacy concerns hinder the research to present robust solutions. To address these challenges, we propose a framework for automated cybercrime complaint classification. The framework leverages Hinglish-adapted transformers, such as HingBERT and HingRoBERTa, to handle code-mixed inputs effectively. We employ the real-world dataset provided by Indian Cybercrime Coordination Centre (I4C) during CyberGuard AI Hackathon 2024. We employ GenAI open source model-based data augmentation method to address class imbalance. We also employ privacy-aware preprocessing to ensure compliance with ethical standards while maintaining data integrity. Our solution achieves significant performance improvements, with HingRoBERTa attaining an accuracy of 74.41% and an F1-score of 71.49%. We also develop ready-to-use tool by integrating Django REST backend with a modern frontend. The developed tool is scalable and ready for real-world deployment in platforms like the National Cyber Crime Reporting Portal. This work bridges critical gaps in cybercrime complaint management, offering a scalable, privacy-conscious, and adaptable solution for modern cybersecurity challenges.
Figures
Reference graph
Works this paper leans on
-
[1]
H. Ding, J. Zhai, Y . Nan, and S. Ma, ” {AIRTAG}: Towards Automated Attack Investigation by Unsupervised Learning with Log Texts,” in Proceedings of the 32nd USENIX Security Symposium (USENIX Security 23), 2023, pp. 373–390
work page 2023
-
[2]
Learning from imbalanced data,
H. He and E. A. Garcia, “Learning from imbalanced data,” IEEE Transactions on Knowledge and Data Engineering , vol. 21, no. 9, pp. 1263–1284, 2009
work page 2009
-
[3]
Measuring sentiment annotation quality in code-mixed social media content: Some observations,
A. Joshi, P. Bhatia, P. Bhattacharyya, and M. Carman, “Measuring sentiment annotation quality in code-mixed social media content: Some observations,” in Proceedings of the 10th International Conference on Language Resources and Evaluation (LREC) , 2016
work page 2016
-
[4]
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (NAACL-HLT) , 2019
work page 2019
-
[5]
RoBERTa: A Robustly Optimized BERT Pretraining Approach,
Y . Liu et al., “RoBERTa: A Robustly Optimized BERT Pretraining Approach,” arXiv preprint arXiv:1907.11692 , 2019
arXiv 1907
-
[6]
TTPXHunter: Action- able threat intelligence extraction as TTPs from finished cyber threat reports,
N. Rani, B. Saha, V . Maurya, and S. K. Shukla, “TTPXHunter: Action- able threat intelligence extraction as TTPs from finished cyber threat reports,” Digital Threats: Research and Practice , ACM, 2024
work page 2024
-
[7]
Financial fraud: A literature review,
A. Reurink, “Financial fraud: A literature review,” Journal of Economic Surveys, vol. 32, no. 5, pp. 1292–1325, 2018
work page 2018
-
[8]
L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models
R. Nayak and R. Joshi, “L3Cube-HingCorpus and HingBERT: A code mixed Hindi-English dataset and BERT language models,” arXiv preprint arXiv:2204.08398, 2022
work page Pith review arXiv 2022
Show all 33 references
-
[9]
A new dataset for natural language inference from code-mixed conversations,
S. Khanuja, A. Dhingra, R. Kumar, and M. M. Khapra, “A new dataset for natural language inference from code-mixed conversations,” in Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2020
2020
-
[10]
CAMELON: A System for Crime Metadata Extraction and Spatiotemporal Visualization from Online News Articles,
S. Pongpaichet, B. Sukosit, C. Duangtanawat, J. Jamjongdamrongkit, C. Mahacharoensuk, K. Matangkarat, P. Singhajan, T. Noraset, and S. Tuarob, “CAMELON: A System for Crime Metadata Extraction and Spatiotemporal Visualization from Online News Articles,” IEEE Access, 2024
2024
-
[11]
Times of India, ”India saw 129 cybercrimes per lakh population in 2023,” Available at: https://timesofindia.indiatimes.com/india/india -saw-129-cybercrimes-per-lakh-population-in-2023/articleshow/1065 24847.cms, Accessed: 11-12-2024
2023
-
[12]
Rahma and A
F. Rahma and A. Romadhony, ”Rule-based crime information extraction on Indonesian digital news,” in 2021 International Conference on Data Science and Its Applications (ICoDSA) , 2021, pp. 10–15
2021
-
[13]
TTPHunter: Automated extraction of actionable intelligence as TTPs from narrative threat reports,
Nanda Rani, Bikash Saha, Vikas Maurya, and Sandeep Kumar Shukla, “TTPHunter: Automated extraction of actionable intelligence as TTPs from narrative threat reports,” Proceedings of the 2023 Australasian Computer Science Week, pp. 126–134, 2023
2023
-
[14]
Text Classification on Cybercrime Cases From News Articles Using Supervised Learning
N. M. F. N. M. Nizam, S. Mutalib, M. Y . Darus, A. Ismail, H. Mokayed, and S. Abdul-Rahman, “Text Classification on Cybercrime Cases From News Articles Using Supervised Learning.”
-
[15]
Leveraging machine learning for ran- somware detection,
N. Rani and S. V . Dhavale, “Leveraging machine learning for ran- somware detection,” arXiv preprint arXiv:2206.01919 , 2022
2022 arXiv
-
[16]
The future of financial fraud,
J. M. Karpoff, “The future of financial fraud,” Journal of Corporate Finance, vol. 66, p. 101694, 2021
2021
-
[17]
Business Standard, ”Here is how much Indians lost to cyber frauds between Jan and Apr of 2024,” Available at: https://www.business -standard.com/india-news/here-is-how-much-indians-lost-to-cyber-f rauds-between-jan-and-apr-of-2024-124052700151 1.html, Accessed: 10-12-2024
2024
-
[18]
Identification and classification of cybercrimes using text mining technique,
S. Andleeb, R. Ahmed, Z. Ahmed, and M. Kanwal, “Identification and classification of cybercrimes using text mining technique,” in 2019 International Conference on Frontiers of Information Technology (FIT) , IEEE, 2019, pp. 227–2275
2019
-
[19]
Hassanat, Ahmad S
Ahmad B. Hassanat, Ahmad S. Tarawneh, Ghada A. Altarawneh, and Abdullah Almuhaimeed, ”Stop oversampling for class imbalance learning: A critical review,” arXiv preprint arXiv:2202.03579 , 2022
2022 arXiv
-
[20]
Machine learning techniques applied to Bangla crime news classification,
N. Islam, R. Siddiqua, and S. Momen, “Machine learning techniques applied to Bangla crime news classification,” in Proceedings of the 2022 IEEE 2nd Conference on Information Technology and Data Science (CITDS), pp. 130–135, 2022
2022
-
[21]
Norton, ”Cybersecurity Statistics,” Available at: https://us.norton.com/ blog/emerging-threats/cybersecurity-statistics, Accessed: 11-12-2024
2024
-
[22]
Code mixing: A computational study of Hindi-English switching,
S. Bhat, M. Choudhury, and K. Bali, “Code mixing: A computational study of Hindi-English switching,” in Proceedings of ACL, 2018
2018
-
[23]
Contextual augmentation: Data augmentation by words with paradigmatic relations,
S. Kobayashi, “Contextual augmentation: Data augmentation by words with paradigmatic relations,” arXiv preprint arXiv:1805.06201 , 2018
2018 arXiv
-
[24]
Imbalanced text classification: A critical review,
S. Karimi et al., “Imbalanced text classification: A critical review,” arXiv preprint arXiv:2103.11367, 2021
2021 arXiv
-
[25]
A brief review of new threats and countermeasures in digital crime and cyber terrorism,
M. Dawson, “A brief review of new threats and countermeasures in digital crime and cyber terrorism,” New Threats and Countermeasures in Digital Crime and Cyber Terrorism , pp. 1–7, 2015
2015
-
[26]
A survey on machine learning-based ransomware detection,
N. Rani, S. V . Dhavale, A. Singh, and A. Mehra, “A survey on machine learning-based ransomware detection,” in Proceedings of the Seventh International Conference on Mathematics and Computing: ICMC 2021 , Springer, pp. 171–186, 2022
2021
-
[27]
Anonymisation and pseudonymisation of text data: A survey of practices, techniques, and algorithms,
P. Lison and E. Pettersen, “Anonymisation and pseudonymisation of text data: A survey of practices, techniques, and algorithms,” arXiv preprint arXiv:1911.11344, 2019
1911 arXiv
-
[28]
Cyber Complaint Automation System,
A. V . Prabhu, M. J. Jefiya, J. D. Joseph, T. Sunny, and C. M. Abraham, “Cyber Complaint Automation System,” in 2023 Ad- vanced Computing and Communication Technologies for High Perfor- mance Applications (ACCTHPA) , 2023, pp. 1–5, doi: 10.1109/ACC- THPA57160.2023.10083354
2023
-
[29]
A comprehensive survey of advanced persistent threat attribution: Taxonomy, methods, challenges and open research problems,
N. Rani, B. Saha, and S. K. Shukla, “A comprehensive survey of advanced persistent threat attribution: Taxonomy, methods, challenges and open research problems,” arXiv preprint arXiv:2409.11415 , 2024
2024 arXiv
-
[30]
Songailait ˙e, E
M. Songailait ˙e, E. Kankevi ˇci¯ut˙e, B. Zhyhun, and J. Mandravickait ˙e, ”BERT-Based Models for Phishing Detection,” in CEUR Workshop Pro- ceedings: IVUS 2023: Proceedings of the 28th International Conference 10 on Information Society and University Studies, Kaunas, Lithuani...
2023
-
[31]
Distinguishing between facts and opinions for sentiment analysis: Survey and chal- lenges,
I. Chaturvedi, E. Cambria, R. E. Welsch, and F. Herrera, “Distinguishing between facts and opinions for sentiment analysis: Survey and chal- lenges,” Information Fusion, vol. 44, pp. 65–77, 2018
2018
-
[32]
Machine learning approach for text classification in cybercrime,
S. Kumari, Z. Saquib, and S. Pawar, “Machine learning approach for text classification in cybercrime,” in 2018 Fourth International Conference on Computing Communication Control and Automation (ICCUBEA) , 2018, pp. 1–6
2018
-
[33]
Information extraction from Indonesian crime news with named entity recognition,
R. R. Sedik and A. Romadhony, “Information extraction from Indonesian crime news with named entity recognition,” in Proceedings of the 2023 15th International Conference on Knowledge and Smart Technology (KST), 2023, pp. 1–5, IEEE
2023
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.