{"id":"bff7845b-2223-4d85-ba1f-5952d6026e77","arxiv_id":"2412.16614","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"HingRoBERTa, fine-tuned on augmented Hinglish cybercrime complaints, reaches 74.41% accuracy and 71.49% F1, outperforming generic BERT/RoBERTa and TF-IDF baselines.","lead":"An automated system using Hinglish-adapted transformers (HingRoBERTa) classifies cybercrime complaints written in code-mixed Hinglish, achieving 74.41% accuracy and 71.49% F1 on a real-world Indian dataset. A generalist should read it to judge whether language-specialized NLP plus LLM data augmentation is a viable path for scalable, privacy-conscious law-enforcement triaging.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 74.41% accuracy/F1 are computed on a test set from which entire categories (e.g., Crime Against Women & Children, Report Unlawful Content) are excluded (Sec. III-C), so the metric does not cover the full complaint distribution and cannot support the 'ready for real-world deployment' claim.","rationale":"The reader chose augmentation label preservation as the weakest assumption; I agree that assumption is unverified, but the more load-bearing issue is that the evaluation itself excludes entire categories. The manuscript explicitly states that classes found only in the test set and rare classes are removed before evaluation (Sec. III-C.1). That exclusion is not hidden, but its effect on the headline metric is not acknowledged in the interpretation. A classifier evaluated without categories it cannot possibly learn will have inflated macro scores relative to the stated 14-category problem, and the paper's strongest claim explicitly includes real-world deployment readiness. Even if every augmented sample were perfectly labeled, the reported metric would still not estimate performance on the full distribution of complaints. This is directly checkable by re-evaluating on the unpruned holdout set. I therefore keep the reader's CONDITIONAL verdict: acceptance should require reporting results on the complete original label set, with per-class breakdowns. The concern is mentioned in the reader's rationale, though not selected as the weakest assumption, so my agreement is partial.","tokens_in":12580,"tokens_out":6137,"duration_ms":55773,"concrete_test":"Re-run the fine-tuned HingRoBERTa on the original I4C holdout test set without dropping Crime Against Women & Children or Report Unlawful Content, and without collapsing labels through the standardization table; report per-class precision, recall, and F1 plus macro/weighted averages over the full original label set. If accuracy/F1 drops materially or the removed categories are classified at chance, the 74.41% claim is specific to the pruned set and the deployment-readiness claim must be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is in the evaluation protocol, not the model. Section III-C.1 states that categories absent from the training set, such as Crime Against Women & Children found only in the test set, are excluded from evaluation, and classes with fewer than 2 training samples, such as Report Unlawful Content, are removed. Table VI is therefore computed on a pruned label set, not on the full set of complaint categories that a deployed system would encounter. If a complaint belongs to a removed category, the model cannot emit the correct label, yet such failures never enter the reported 74.41% accuracy or 71.49% F1. This makes the headline numbers unrepresentative of the National Cyber Crime Reporting Portal scenario the paper claims readiness for. The label standardization in Table III also merges distinctions among original categories; if test labels are standardized the same way, the metric is further inflated relative to the original task. Section V-C mentions a no-augmentation control but does not report its numerical results, so the claimed contribution of augmentation is also quantitatively unsupported. The reader's augmentation-label-preservation concern is real, but the test-set pruning is more fundamental: it changes what is being measured before any training-data assumption is considered.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an automated pipeline for classifying Hinglish cybercrime complaints, combining privacy-aware preprocessing, LLaMA-based data augmentation for class imbalance, and fine-tuned transformer models (BERT, RoBERTa, HingBERT, HingRoBERTa). Using a real-world dataset from the I4C CyberGuard AI Hackathon, the authors report that HingRoBERTa achieves the highest accuracy (74.41%) and F1-score (71.49%). A Django-based deployment is described as ready for real-world use on platforms like the National Cyber Crime Reporting Portal.","tokens_in":12778,"tokens_out":2110,"duration_ms":18636,"significance":"If the reported results are reliable, this work addresses a genuine operational need: automated triage of code-mixed (Hinglish) cybercrime complaints while respecting privacy constraints. The paper also contributes an augmented dataset (planned release) and a practical deployment framework. The comparison of generic vs. Hinglish-adapted transformers on a private, real-world dataset is valuable for the NLP-for-security community. However, the evaluation protocol has several support gaps that must be addressed before the central deployment claim can be accepted.","major_comments":[{"comment":"The reported accuracy (74.41%) and F1-score (71.49%) are computed on a test set from which entire categories are excluded: classes absent from the training set (e.g., Crime Against Women & Children) and classes with fewer than 2 training samples (e.g., Report Unlawful Content) are removed before evaluation. A deployed system would encounter complaints from these categories, and because the model cannot emit those labels, such failures are never reflected in the headline metrics. The paper should either evaluate on the full test set (including all original 14 categories) or explicitly report the number of excluded samples and provide per-class results for the excluded categories (e.g., as an 'other' class or with a not-classified rate). Without this, the claim of 'ready for real-world deployment' in Section V-D and the abstract is not supported.","section":"III-C.1 and Table VI"},{"comment":"Section V-C states that a controlled experiment was conducted by training HingRoBERTa without augmentation and that the resulting model showed weaker performance, 'especially on minority classes,' but no numerical results are reported. This is a load-bearing claim because the paper attributes a significant part of the improvement to augmentation. The authors should provide a table comparing HingRoBERTa with and without augmentation, including accuracy, F1, and per-class recall for minority categories, so readers can quantitatively assess the contribution of augmentation.","section":"V-C"},{"comment":"The augmentation pipeline uses LLaMA 3.1-7b to paraphrase original complaints and retains only samples with a BERTScore similarity above 97%. However, the threshold filters for semantic similarity to the source text, not for preservation of the original crime category label. There is no manual or automated verification that the generated paraphrases indeed belong to the same class. Since augmentation is applied per-class to balance the distribution, mislabeled augmented samples could inflate minority-class metrics. The authors should add a label-verification step (e.g., human evaluation on a sample, or a secondary classifier's label agreement) and report the results.","section":"III-D"},{"comment":"Each model is evaluated on a single run, with no confidence intervals, error bars, or significance tests. The reported differences are small (e.g., F1 71.02 for HingBERT vs. 71.49 for HingRoBERTa), and without variance estimates it is unclear whether the observed ordering is reliable. At minimum, the authors should perform multiple runs (e.g., 3–5 seeds) and report mean and standard deviation, and ideally a paired significance test (e.g., McNemar's test) for the top-performing models.","section":"V-B and Table VI"}],"minor_comments":[{"comment":"The dataset size is written as '85, 8754' in the text; this appears to be a typo for '85,875'.","section":"III-B"},{"comment":"The similarity metric is referred to as 'BERTF1-score' but the described method (cosine similarity over BERT embeddings) is BERTScore; please use the standard terminology consistently.","section":"III-D"},{"comment":"The label standardization dictionary should state explicitly whether the same mapping is applied to both training and test labels. If test labels are standardized while the original task used the finer-grained labels, the reported metrics reflect a modified task rather than the original complaint categories.","section":"Table III"},{"comment":"References [6] and [13] appear to refer to the same work (TTPXHunter) but with different titles and venues; please unify or disambiguate.","section":"References"},{"comment":"The related work comparison in Table I omits dataset size for Prabhu et al. ('−') and does not list the number of classes; consider adding these details for a fairer comparison.","section":"II and Table I"}],"recommendation":"major_revision","confidential_remarks":"The paper's central empirical claim is plausible, but the evaluation protocol has gaps that need to be closed: test-set pruning, missing augmentation-control numbers, no label-preservation check for augmented data, and no statistical significance. The claim of 'ready for real-world deployment' is currently overstrong given that the model cannot classify entire categories that appear in the test distribution. The dataset is private and not yet released, which limits reproducibility, but the authors have stated an intention to release it upon acceptance. I recommend major revision rather than rejection because the identified issues are addressable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short take: this is a competent applied NLP paper that deserves a serious look, but the reported numbers are not as solid as they look.\n\nWhat's new: the paper applies Hinglish-adapted transformers (HingBERT, HingRoBERTa) to a real-world dataset of 85k+ Hinglish cybercrime complaints from the I4C hackathon, with privacy-aware preprocessing (entity anonymization) and GenAI-based augmentation using LLaMA 3.1-7b with BERTScore filtering. That combination is new for this domain, and the dataset itself, if released, would be a useful resource. The comparison against TF-IDF baselines, generic BERT/RoBERTa, and Hinglish variants is a reasonable evaluation design, and the anonymization step is a genuine practical plus.\n\nThe soft spots are mostly in the evaluation protocol. The biggest one: Section III-C1 excludes categories absent from the training set (e.g., Crime Against Women & Children) and classes with fewer than 2 training samples (e.g., Report Unlawful Content) from evaluation. The 74.41% accuracy and 71.49% F1 are therefore computed on a pruned label set. A deployed system on the National Cyber Crime Reporting Portal would have to handle exactly those categories, and the paper's 'ready for real-world deployment' claim isn't supported by numbers that ignore them. The label standardization in Table III further merges overlapping categories, which could inflate the metrics relative to the original 14-class task.\n\nAlso, there's a single run per model with no error bars or significance tests; the differences between HingRoBERTa and HingBERT (about 1.5 accuracy points, 0.5 F1) could be noise. The augmentation control experiment is mentioned but never quantified, so the claimed benefit of augmentation is unverified. The dataset, code, and tool are not released, which blocks replication.\n\nNone of this is fatal to the paper's core idea. The pipeline is coherent and the privacy preprocessing is a worthwhile contribution. But the evaluation needs to be redone or substantially revised before the central claim holds. A referee should ask for: metrics on the full label set (or a clear justification for exclusion), confidence intervals or multiple runs, numeric results for the no-augmentation baseline, and a commitment to release artifacts or at least a data sample.\n\nWho it's for: researchers and practitioners in code-mixed NLP or cybercrime triage. It's a solid engineering paper with a valuable dataset, but not a methodological breakthrough. Send it to peer review, but expect heavy revision.","headline":"Useful applied paper with a real dataset and a sensible privacy-aware pipeline, but the headline metrics are computed on a pruned label set, so the deployment claim needs scrutiny.","tokens_in":13387,"tokens_out":3154,"would_cite":false,"duration_ms":27031,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HingRoBERTa, a Hinglish-adapted transformer, reaches 74.41% accuracy and 71.49% F1 on real-world cybercrime complaints, and the paper claims its pipeline is ready for deployment.","keywords":["cybercrime classification","Hinglish","code-mixed text","transformer language models","HingRoBERTa","data augmentation","privacy-aware NLP","imbalanced classification"],"falsifier":"Manually relabel a random sample of the augmented complaints and retrain HingRoBERTa with corrected labels; if many augmented samples carry a different crime type than their source, and minority-class F1 drops under corrected training, the reported augmentation benefit is partly an artifact of label drift.","tokens_in":12327,"feed_emoji":"🛡️","tokens_out":12923,"duration_ms":93476,"temperature":0.7,"pith_summary":"The paper sets out to show that automated triage of cybercrime complaints written in Hinglish, the code-mixed Hindi-English used widely in India, is best performed by a transformer model pretrained on Hinglish and fine-tuned on a real-world, anonymized complaint dataset. Using a government-supplied dataset of about 85,875 complaints across 14 crime categories, the authors report that HingRoBERTa reaches 74.41% accuracy and a 71.49% F1-score, surpassing generic BERT and RoBERTa, the Hinglish-adapted HingBERT, and TF-IDF-based machine-learning baselines. They attribute the result to three coordinated components: Hinglish-specific pretraining, generative augmentation of minority classes with LLaMA 3.1-7b paraphrase generation filtered by a 97% similarity threshold, and privacy-preserving replacement of personal entities with placeholders. The authors further claim that this improvement, though a few percentage points, is meaningful for real-world law-enforcement operations and that the accompanying deployed tool is ready for integration into national cybercrime reporting platforms.","feed_headline":"HingRoBERTa tops cybercrime complaint classification at 74.41%","feed_subtitle":"It is fine-tuned on real-world Hinglish complaints, synthetic minority-class examples, and anonymized personal data.","key_machinery":"The load-bearing component is HingRoBERTa, a Hinglish-adapted RoBERTa model that produces contextual embeddings for code-mixed complaint text and feeds a linear classifier over 14 standardized crime-type labels. Around this model sits the augmentation machinery: LLaMA 3.1-7b generates paraphrases of minority-class complaints after entity replacement, and a similarity filter with a 97% threshold discards sentences that drift from the original meaning; this takes the dataset from 85,875 to 109,294 samples while rebalancing rare categories such as ransomware and cyber trafficking. Privacy preprocessing—regex-based plus statistical entity recognition that replaces names, phones, emails, addresses, and monetary values with placeholder tokens—is applied before both augmentation and training, so the model learns from semantic patterns rather than memorizing individuals.","core_discovery":"On the paper's own terms, the central discovery is that HingRoBERTa—a RoBERTa variant pretrained on code-mixed Hindi-English text—is the best classifier for cybercrime complaints among the models tested, achieving 74.41% accuracy and 71.49% F1-score on a highly imbalanced real-world dataset of Hinglish complaints spanning 14 standardized crime types. The paper argues this result validates the importance of language adaptation, since both Hinglish-adapted models (HingBERT and HingRoBERTa) beat their generic counterparts (BERT and RoBERTa). It also claims that the GenAI-based augmentation pipeline materially improves minority-class performance: a controlled experiment without augmentation showed weaker results, especially on underrepresented categories. The paper's conclusion is that the full pipeline of language adaptation, generative rebalancing, and anonymization yields a privacy-conscious classifier that is deployable in operational settings such as national reporting portals.","pith_inferences":["A testable consequence of the augmentation design is that label drift could inflate minority-class F1: the 97% similarity filter checks wording closeness, not crime-category agreement, so a manual audit of augmented samples would settle whether the reported gains are real.","Because the paper reports only aggregate metrics, per-class precision and recall on a human-verified test set would show whether the headline accuracy is driven mainly by the dominant financial-fraud class.","The same pipeline should transfer to other code-mixed Indian languages (Banglish, Tanglish) by swapping HingRoBERTa for a matching language-adapted transformer and rerunning the augmentation and anonymization steps.","An ablation against conventional oversampling with matched class counts would isolate whether the gains come from semantic diversity in paraphrases or simply from rebalancing."],"forward_implications":["For Indian cybercrime complaint triage, Hinglish-adapted transformers should be preferred over generic English-only models, because HingRoBERTa outperformed BERT and RoBERTa on the same data.","Law-enforcement agencies can scale triage by fine-tuning this pipeline on their own complaint streams, since the preprocessing, augmentation, and deployment components are modular.","Minority crime categories such as ransomware and cyber trafficking can be reinforced without manual data collection, through LLM paraphrase augmentation of scarce examples.","Anonymization can be built into the training pipeline without sacrificing accuracy, which matters for compliance with data-protection rules.","The model's performance provides a baseline for future work on code-mixed complaint classification, including hierarchical or finer-grained crime labels."],"supporting_citations":[{"why":"Supplies the Hinglish-pretrained HingBERT and HingRoBERTa models whose advantage the paper tests.","marker":"[8]"},{"why":"Defines BERT, the generic transformer baseline the paper's models must beat.","marker":"[4]"},{"why":"Defines RoBERTa, the generic baseline and the base architecture HingRoBERTa adapts.","marker":"[5]"},{"why":"Provides the contextual augmentation concept that the LLaMA-based paraphrase pipeline extends.","marker":"[23]"},{"why":"Makes the case against naive oversampling for imbalanced text, motivating generative augmentation.","marker":"[24]"},{"why":"Documents the distribution-distortion risks of oversampling, supporting the same choice.","marker":"[19]"},{"why":"Motivates the anonymization step by surveying practices for removing personal information from text.","marker":"[27]"}],"fun_headline_variants":["HingRoBERTa beats generic BERT on Hinglish cybercrime complaints","74.41% accuracy: HingRoBERTa classifies Hinglish cybercrime complaints","GenAI augmentation improves Hinglish cybercrime minority-class F1","HingRoBERTa outclasses BERT on code-mixed cybercrime complaints","Anonymized Hinglish complaints train HingRoBERTa for cybercrime triage"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LLM-generated paraphrases preserve the original crime label—the similarity filter checks wording closeness, not label correctness—so if mislabeled paraphrases enter training, the reported gains on rare crime types could be inflated.","fun_headline_variants_meta":{"raw":{"variants":["HingRoBERTa beats generic BERT on Hinglish cybercrime complaints","74.41% accuracy: HingRoBERTa classifies Hinglish cybercrime complaints","GenAI augmentation improves Hinglish cybercrime minority-class F1","HingRoBERTa outclasses BERT on code-mixed cybercrime complaints","Anonymized Hinglish complaints train HingRoBERTa for cybercrime triage"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000612,"raw_usage":{"total_tokens":2880,"prompt_tokens":1011,"completion_tokens":1869,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":627,"completion_tokens_details":{"reasoning_tokens":1761}},"tokens_in":627,"tokens_out":1869,"duration_ms":13309,"temperature":1.0,"reasoning_tokens":1761,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:24:40.223254+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually relabel a random sample of the augmented complaints and retrain HingRoBERTa with corrected labels; if many augmented samples carry a different crime type than their source, and minority-class F1 drops under corrected training, the reported augmentation benefit is partly an artifact of label drift.","supporting_citations":[{"cited_title":"L3Cube-HingCorpus and HingBERT: A Code Mixed Hindi-English Dataset and BERT Language Models","cited_arxiv_id":"2204.08398","evidence_quote":"Supplies the Hinglish-pretrained HingBERT and HingRoBERTa models whose advantage the paper tests."},{"cited_title":"BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,","cited_arxiv_id":null,"evidence_quote":"Defines BERT, the generic transformer baseline the paper's models must beat."},{"cited_title":"ROSITA: Refined BERT cOmpreSsion with InTegrAted techniques","cited_arxiv_id":"2103.11367","evidence_quote":"Makes the case against naive oversampling for imbalanced text, motivating generative augmentation."}],"review_version":1}