{"id":"72f8886e-f51e-4479-895e-d2f8e38d063c","arxiv_id":"2412.08163","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"An ensemble of XLM-RoBERTa, MuRIL, and an abusive-tuned MuRIL achieved 0.7762 recall and 0.6914 F1 on Devanagari hate speech detection.","lead":"Researchers tested several multilingual BERT models and an ensemble for detecting hate speech in Hindi and Nepali tweets written in Devanagari script. Their best ensemble placed 3rd out of 31 teams on recall for the CHIPSAL@COLING 2025 shared task, with a moderate F1 score.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ensemble construction reads test-set metrics in Table 3, so the recall gain is partly a selection artifact; the OR rule trivially guarantees recall non-decrease.","rationale":"The reader's conditional verdict is appropriate, but the most load-bearing weakness is not the augmentation label-preservation issue. The augmentation concern is real and acknowledged in the paper's Limitations, yet it affects all trained models roughly equally and does not specifically explain why the ensemble outperforms individual models in recall. The sharper problem is that the ensemble was constructed by reading test-set metrics in Table 3, and the cascade is logically an OR of the three models. Since an OR ensemble has recall at least as high as its best member, the observed recall improvement is guaranteed by construction; the only informative question is the precision/F1 trade-off, and the ensemble actually loses F1 relative to M7. The external Codalab evaluation is credible evidence that the submitted predictions scored as reported, and the code release is a positive. However, the paper's interpretive claim that the ensemble 'demonstrates' an improvement over single models is not supported by an unbiased comparison. A validation-based selection protocol would settle whether the improvement generalizes or is a selection artifact. This reinforces the existing CONDITIONAL verdict rather than changing it.","tokens_in":8167,"tokens_out":6858,"duration_ms":72186,"concrete_test":"Hold out a validation split (e.g., 20% of the official training set or the phase-2 development set) and choose the cascade members and any threshold using only validation recall/F1; then evaluate the fixed cascade on the official test set exactly once. If ensemble test recall no longer exceeds the best single-model test recall by a comparable margin, the reported improvement is a test-set selection artifact. As a secondary check, compute recall and precision of the union of M7 and M3 only, to see whether the M1 fallback adds any true positives beyond that union.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.3 states that M7, M3, and M1 were chosen as primary, secondary, and fallback 'based on each model's unique strengths' using Table 3, which reports test-set results. The cascade rule is equivalent to predicting hate if any of M7, M3, or M1 predicts hate, so the ensemble's positive set is the union of the three models' positive predictions. Consequently, ensemble recall is at least the maximum individual recall by construction; the observed gain from 0.7381 to 0.7762 is not evidence that ensembling adds signal beyond taking a union. Worse, because the models were selected after seeing test labels, the reported ensemble numbers are optimistically biased relative to any pre-registered rule. The paper's own Table 3 shows ensemble F1 (0.6914) is slightly lower than M7's F1 (0.6933), so calling it the 'best performing model' is at best selective. None of this impugns the external leaderboard score, but it undercuts the paper's claim that the ensemble 'demonstrates' improved recall. A validation-based selection protocol is needed before the improvement can be attributed to the ensemble rather than to test-set selection.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper describes a hate speech detection system for Devanagari-scripted Hindi and Nepali, developed for Subtask B of the CHIPSAL@COLING 2025 shared task. The authors fine-tune eight models (MuRIL, XLM-RoBERTa, IndicBERT, and FastText-based variants), apply backtranslation with cosine-similarity filtering for data augmentation, and combine three models (M7, M3, M1) into a priority-based cascade. On the official test set they report recall 0.7762 (rank 3/31), precision 0.6639, F1 0.6914 (rank 17/31), and accuracy 0.8258. The central claim is that this ensemble improves recall over any single model for low-resource Devanagari hate speech detection.","tokens_in":8399,"tokens_out":5558,"duration_ms":51775,"significance":"If the reported results are taken at face value, the paper is a useful system description for a shared task: it compares eight architectures on a low-resource Devanagari hate speech task, uses an official external leaderboard for evaluation, and makes code publicly available. The finding that a multilingual BERT cascade achieves high recall (rank 3 of 31) is of practical interest for recall-oriented moderation scenarios. However, the central claim that the ensemble 'demonstrates' improved recall is weakened by two load-bearing issues: the ensemble members and ordering are selected using the test-set metrics in Table 3, and the cascade rule is logically an OR of three models, so part of the recall gain is guaranteed by construction. The paper also contains a numerical inconsistency in the augmentation accounting and no uncertainty estimates. With a validation-based selection protocol and reframed claims, the contribution would be solid.","major_comments":[{"comment":"The ensemble construction is selected on the test set. Section 4.3 states that M7, M3, and M1 were chosen as primary, secondary, and fallback 'based on each model's unique strengths' using Table 3, which reports test-set results. This means the reported ensemble metrics are optimistically biased relative to any pre-specified rule, because the model ordering was chosen after seeing the test labels. Moreover, the cascade rule is equivalent to predicting hate if any of M7, M3, or M1 predicts hate, so the ensemble recall is at least the maximum individual recall by construction; the increase from 0.7381 to 0.7762 is not by itself evidence that ensembling adds signal beyond taking the union of positive predictions. The paper should select the ensemble on a held-out development split and report the corresponding test metrics, and should compare the cascade against a plain OR of the three models or against M7 alone. Note also that Table 3 shows the ensemble F1 (0.6914) is slightly lower than M7's F1 (0.6933), so calling the ensemble the 'best performing model' in the abstract is at best selective.","section":"Section 4.3, Table 3"},{"comment":"The data augmentation accounting is internally inconsistent. Table 1 lists 2,214 training hate instances (679 Hindi + 1,535 Nepali) and 474 evaluation hate instances (142 Hindi + 332 Nepali). The text states that 'the training set grew to 13,695 instances by incorporating the original 2,214 training, 474 evaluation, and their augmented instances.' Starting from the Table 1 training total of 19,019, no combination of duplication and backtranslation of the hate subset yields 13,695. The paper should give exact counts of original, duplicated, and backtranslated examples per language, and clarify whether the 13,695 figure includes the non-hate training instances. As written, the data setup is not reproducible.","section":"Section 4.1, Table 1"},{"comment":"No repeated runs, seeds, or confidence intervals are reported; all model comparisons are point estimates on the test set. Given the small hate class (142 Hindi and 332 Nepali evaluation instances in Table 1), differences such as M7 recall 0.7381 versus M3 recall 0.6877 could be within noise. The paper should provide variance estimates from multiple fine-tuning runs on a development set, or at minimum explicitly state that the observed ordering is not statistically evaluated. Currently the model ranking in Table 3 is asserted without uncertainty quantification.","section":"Section 4.4, Table 3"},{"comment":"The augmentation pipeline assumes that backtranslated examples filtered by cosine similarity greater than 0.9 retain their hate labels. The Limitations section explicitly concedes that backtranslation can shift formal or respectful Nepali terms to informal registers and alter sentiment, so the cosine threshold does not guarantee label preservation. Because these augmented examples are used to train all models, the measured performance could reflect artifacts of the augmentation procedure rather than hate detection ability. The paper should add a small manual or automatic label-preservation check on the augmented samples, and should describe how the 0.9 threshold was chosen without reference to the test-set results.","section":"Section 4.1, Limitations"}],"minor_comments":[{"comment":"The text says 'The First Workshop on South East Asian Language Processing' but the cited workshop (Sarveswaran et al., 2025) is on South Asian languages; the region should be corrected.","section":"Introduction"},{"comment":"References 7 and 8 (Conneau et al., 2019 and 2020) appear to be the same paper; they should be merged or clearly distinguished.","section":"References"},{"comment":"The phrase 'an token embedding-based model' should read 'a token embedding-based model.'","section":"Table 2"},{"comment":"The cascade rule could be written more transparently as an OR of M7, M3, and M1; the current formulation obscures the fact that the ensemble's positive set is the union of the three models' positive predictions.","section":"Section 4.3"},{"comment":"Figure 1 is referenced in Section 3 but no figure content is visible in the manuscript; please ensure the workflow figure is included in the final version.","section":"Figure 1"},{"comment":"The caption refers to dark green and dark red cells for best and worst performance, but the printed table is monochrome; add explicit markers such as boldface or asterisks so the best and worst values are identifiable.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":"This is a shared-task system description with an externally evaluated leaderboard score, which gives it some value regardless of the internal analysis issues. The main blocker is the test-set-based ensemble selection combined with the OR structure of the cascade; this needs to be fixed by a validation-based protocol or by substantially softening the claims. The numerical inconsistency in Section 4.1 is easy to fix but must be addressed. I do not see grounds for rejection, since the official evaluation is external and the code is available; the paper is suitable for major revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real news: this is a solid, modest shared-task system description, not a methods paper. The only genuinely new result is the official test-set performance of a three-model cascade (XLM-R, MuRIL, MuRIL-abusive) on the CHIPSAL@COLING 2025 Devanagari hate speech subtask: recall 0.7762, rank 3 of 31. The code is public and the evaluation was run on Codalab, so the leaderboard numbers are externally anchored. That is real value for anyone needing a baseline in Hindi/Nepali hate speech.\n\nWhat the paper does well: it tries several architectures, reports all of them in one table, and the Limitations section is unusually honest. It explicitly concedes that backtranslation can shift formal Nepali terms (tapai/hajur to timi) and alter sentiment, which directly undermines the label-preservation assumption of the augmentation in Section 4.1. That kind of admission should be credited.\n\nThe soft spots are real but not fatal. The ensemble is chosen after looking at Table 3, which reports test-set metrics; the priority order M7-positive, then M3, then M1 is a selection artifact, not a pre-registered rule. The cascade is also an OR over three positive predictions, so ensemble recall is guaranteed to be at least the max individual recall; the gain from 0.7381 to 0.7762 doesn't demonstrate that ensembling adds signal. The ensemble F1 (0.6914) is actually lower than M7 alone (0.6933), so calling it the 'best performing model' in the abstract is selective. There are also no repeated runs or error bars, and the cosine-similarity threshold (0.9) is hand-tuned without a validation split. The reader's stress test is right: the recall gain is partly a test-set optimization artifact.\n\nThat said, the external leaderboard score is defensible as a system result, and the paper doesn't overclaim beyond its limitations section. For a shared task, this deserves a serious referee and publication as a system description after the framing is fixed. I'd recommend conditional accept: make clear the ensemble selection used test metrics, remove the 'best performing' language, and add a validation-based analysis if possible.","headline":"A modest, honest shared-task system description whose headline ensemble result is partly a test-set selection artifact, but still a useful external benchmark for Devanagari hate speech.","tokens_in":8915,"tokens_out":2172,"would_cite":true,"duration_ms":21618,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fixed cascade of three fine-tuned multilingual BERT models reaches 0.7762 recall and 0.6914 F1 on the Devanagari hate-speech test set, improving on every single model it is compared with.","keywords":["hate speech detection","Devanagari script","Hindi","Nepali","multilingual BERT","model ensembling","backtranslation","low-resource NLP"],"falsifier":"Have native Hindi and Nepali speakers annotate a sample of the augmented examples that passed the 0.9 cosine filter, and re-run the pipeline without augmentation. If a substantial share of the filtered examples are no longer hate speech, or if the recall gain disappears without augmentation, then the ensemble's improvement is an artifact of duplicated near-identical examples rather than better detection.","tokens_in":7977,"feed_emoji":"🛡️","tokens_out":9418,"duration_ms":85740,"temperature":0.7,"pith_summary":"Devanagari-scripted Hindi and Nepali are low-resource languages for hate speech detection, and this paper tries to show that a deliberately ordered ensemble of three pretrained multilingual transformer models can beat any one of them. The ensemble predicts hate if an XLM-RoBERTa model says hate, otherwise if a MuRIL model says hate, otherwise it returns the prediction of a third MuRIL-based model; on the task's test set this reaches recall 0.7762, precision 0.6639, F1 0.6914, and accuracy 0.8258, with recall ranked third of thirty-one submissions. The paper also argues that backtranslation plus cosine-similarity filtering is a workable response to severe class imbalance, since hate instances were only about one in ten of the training data. The result matters because it is one of the few comparisons of multilingual BERT models and ensembling strategies on a mixed Hindi-Nepali Devanagari hate speech benchmark.","feed_headline":"Three-model cascade lifts hate-speech recall to 0.7762","feed_subtitle":"Ordering multilingual models by catch-rate beats every single model on Hindi and Nepali tweets.","key_machinery":"The load-bearing mechanism is the cascade decision function: output hate if the XLM-RoBERTa model (M7) outputs hate; otherwise output hate if the MuRIL model (M3) outputs hate; otherwise output the decision of the MuRIL-abusive model (M1). It is a deterministic, OR-like composition of three fine-tuned classifiers with no joint training. Each stage is a transformer with its native classification head: M7 is chosen for highest single-model recall, M3 for balanced F1, and M1 for highest precision and accuracy. The cascade works by letting the recall-oriented model vote first, the balanced model catch remaining positives, and the precision-oriented model act as a conservative fallback. This ordering carries the argument because it turns three individually weaker models into a single predictor with higher recall than any of them.","core_discovery":"The central claim is that a three-model cascade, not a learned ensemble but a fixed decision rule, detects hate speech in Devanagari-scripted Hindi and Nepali tweets better than any of the eight individual classifiers evaluated, especially in recall. The rule is: output hate if model M7, an XLM-RoBERTa model, outputs hate; otherwise output hate if model M3, a MuRIL model, outputs hate; otherwise output the decision of model M1, a MuRIL model fine-tuned on abusive-language data. On the official test set the cascade scores 0.7762 recall, 0.6639 precision, 0.6914 F1, and 0.8258 accuracy, placing third of thirty-one by recall and seventeenth by F1. The single best recall model, M7, reaches 0.7381 recall and 0.6933 F1, while the most accurate model, M1, reaches 0.8950 accuracy; the cascade trades a little precision and accuracy for a large recall gain. The authors attribute the improvement to complementary strengths: one model catches the most positives, a second gives balanced coverage, and a high-precision fallback limits false positives when the first two abstain.","pith_inferences":["The cascade's decision rule is equivalent to an OR over the first two models, so alternative orderings, such as putting the high-precision model first, would likely change the precision-recall tradeoff; the paper does not test them.","A direct annotation of backtranslated instances that passed the 0.9 cosine filter would separate the ensemble's genuine linguistic gain from memorization of duplicated examples, and the threshold could be swept to find where label preservation breaks.","Because Hindi and Nepali share the Devanagari script but differ in vocabulary, the language-agnostic embeddings may confuse near-identical words; conditioning the cascade on a language-identification head is a natural extension the paper leaves open.","If language labels for the test set become available, the recall gain could be decomposed per language, since the paper reports only pooled scores and we cannot tell whether the cascade helps Nepali, Hindi, or both."],"forward_implications":["For Devanagari Hindi and Nepali hate speech, fine-tuned multilingual BERT models can be combined by a fixed cascade without any ensemble training, which is a cheap recipe for low-resource classification tasks.","The recall gain from 0.7381 to 0.7762 means roughly four more hate instances per hundred are caught by the ensemble, at the cost of slightly more non-hate tweets being flagged as hate.","XLM-RoBERTa is the strongest single model for recall on this mixed-script dataset, while IndicBERT is weaker, suggesting Nepali text is the bottleneck for Indic-specific models.","Backtranslation with a cosine-similarity filter above 0.9, plus duplication of hate instances, is enough to train competitive transformers despite only 2,214 original hate examples in the training set."],"supporting_citations":[{"why":"Supplies the M1 MuRIL model fine-tuned on abusive-language data, the cascade's high-precision fallback.","marker":"Das et al., 2022"},{"why":"Supplies pretrained MuRIL representations used by M3 and related models for Devanagari Hindi and Nepali.","marker":"Khanuja et al., 2021a"},{"why":"Supplies XLM-RoBERTa, the primary cascade model and the strongest single model for recall.","marker":"Conneau et al., 2020"},{"why":"Supplies the XLM-RoBERTa base model used to compute the cosine similarity that filters backtranslated instances.","marker":"Conneau et al., 2019"},{"why":"Supplies IndicBERT, whose lower performance in the comparison supports the paper's claim about Nepali text.","marker":"Kakwani et al., 2020"},{"why":"Supplies mBART-large-50, the backtranslation engine that creates augmented hate-speech training instances.","marker":"Tang et al., 2020"},{"why":"Defines the shared task, the dataset, and the evaluation metrics the cascade's scores are measured against.","marker":"Thapa et al., 2025"}],"fun_headline_variants":["Cascade of BERTs beats single models on Hindi hate speech","Three-model rule lifts hate recall to 0.7762","Ordered BERTs top recall for Devanagari hate detection","Hindi-Nepali hate speech: fixed cascade wins recall","Backtranslated BERT trio ranks third by recall on hate"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the backtranslated and duplicated hate examples, filtered by cosine similarity above 0.9, still carry their original hate label and teach the model real hate speech rather than translation artifacts; the authors themselves concede that backtranslation can shift respectful Nepali terms to informal register and change sentiment.","fun_headline_variants_meta":{"raw":{"variants":["Cascade of BERTs beats single models on Hindi hate speech","Three-model rule lifts hate recall to 0.7762","Ordered BERTs top recall for Devanagari hate detection","Hindi-Nepali hate speech: fixed cascade wins recall","Backtranslated BERT trio ranks third by recall on hate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000267,"raw_usage":{"total_tokens":1623,"prompt_tokens":964,"completion_tokens":659,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":571}},"tokens_in":580,"tokens_out":659,"duration_ms":6610,"temperature":1.0,"reasoning_tokens":571,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:08:20.954609+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have native Hindi and Nepali speakers annotate a sample of the augmented examples that passed the 0.9 cosine filter, and re-run the pipeline without augmentation. If a substantial share of the filtered examples are no longer hate speech, or if the recall gain disappears without augmentation, then the ensemble's improvement is an artifact of duplicated near-identical examples rather than better detection.","supporting_citations":[{"cited_title":"Khapra, and Pratyush Kumar","cited_arxiv_id":null,"evidence_quote":"Supplies IndicBERT, whose lower performance in the comparison supports the paper's claim about Nepali text."}],"review_version":1}