{"id":"ce0257e9-d23f-4e52-b8e8-cee9df44a757","arxiv_id":"2505.12587","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"CMLFormer, a dual-decoder Transformer with synchronized cross-attention and switching point prediction objectives, improves F1 on HASOC-2021 Hinglish hate speech detection by up to 0.18 over a same-data BERTbase.","lead":"Code-mixed Hinglish text mixes Hindi and English within single sentences, and standard language models struggle with the frequent language switches. This paper proposes CMLFormer, a Transformer with two synchronized decoders and new pre-training objectives that learn switching points, and reports gains on a Hinglish hate speech benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's own numbers contradict the claimed SPP gain: adding SPP to the BiLTM setup drops F1 from 0.431 to 0.295, so the '0.05 through SPP' attribution is unsupported.","rationale":"The reader's weakest assumption focused on the quality of LLM-generated switching labels. That is a real risk, but the more immediate and load-bearing problem is internal: the paper's own Table 1 contradicts the headline attribution. The SPP objective is presented as a source of a 0.05 F1 gain, yet comparing rows 2 and 3 shows SPP reduces F1 by 0.136 once BiLTM is present. Because no SPP-only run exists, the positive comparison to BERTbase cannot separate SPP from the dual-decoder architecture or from BiLTM. Similarly, the 0.18 BiLTM gain is confounded with the architectural difference between CMLFormer and BERTbase. The baseline is also self-trained on a tiny sample rather than the public HingBERT, and no published HASOC-2021 reference numbers are reported, so the external claim 'outperforms known baselines' is not supported. The limitations section honestly acknowledges the small pretraining set and possible overfitting, but it does not address the internal inconsistency in the ablation logic. These issues are correctable in principle, but as written the central claim about the contribution of the two named objectives is not supported by the evidence, which warrants rejection of the current version rather than conditional acceptance.","tokens_in":12509,"tokens_out":3825,"duration_ms":41765,"concrete_test":"Run two additional pre-training/fine-tuning conditions on the same 10k subset and HASOC-2021 split: (a) dual-decoder CMLFormer with MLM+SPP only, no BiLTM; (b) dual-decoder CMLFormer with MLM only, no auxiliary objectives. Fine-tune the encoder in both cases with the same protocol as Table 1, repeat each condition for at least five seeds, and report mean±std F1. If SPP-only does not beat the MLM-only BERTbase, or MLM-only dual-decoder matches the MLM+BiLTM result, the paper's attribution of the gains to BiLTM and SPP fails.","verdict_should_be":"REJECT","load_bearing_attack":"The conclusion says CMLFormer gains 0.18 and 0.05 F1 through BiLTM and SPP, respectively, but Table 1 shows the opposite ordering. Row 2 (MLM+BiLTM) achieves F1=0.431; row 3 (MLM+BiLTM+SPP) achieves F1=0.295. Adding SPP to the BiLTM configuration reduces F1 by 0.136, so SPP cannot be credited with a positive marginal contribution. The only comparison in which the 0.05 appears is against the MLM-only BERTbase row, but no SPP-only condition (MLM+SPP without BiLTM) is run, so the gain cannot be attributed to SPP. The 0.18 BiLTM gain is similarly confounded: every CMLFormer row uses the dual-decoder architecture, while BERTbase does not, and no MLM-only dual-decoder ablation is provided. Thus the effect of BiLTM is not separated from the effect of added decoder parameters and cross-attention. In addition, the baseline is a BERTbase self-trained on the same 10,000-sample subset, not the released HingBERT, and no published HASOC-2021 baselines are cited, so 'outperforms known baselines' is not established. With no error bars or repeated runs, the reported deltas cannot be distinguished from optimization noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CMLFormer, a Transformer with a shared encoder and two synchronously cross-attending decoders, pre-trained on 10,000 sentences sampled from L3Cube-HingCorpus and augmented with Gemini-generated English/Hindi translations and token-level language/switching-point annotations. Six pre-training objectives are combined (MLM, BTSP, BiLTM, TLC, SPP, CMI). After pre-training, the decoders are detached and the encoder is fine-tuned on HASOC-2021 hate-speech detection. Table 1 reports that the MLM+BiLTM configuration reaches F1=0.431 versus F1=0.249 for a locally pre-trained BERTbase, with F1 degrading as further objectives are added; attention visualizations are used to argue that the model attends to switching points. The paper claims F1 gains of 0.18 and 0.05 from BiLTM and SPP, respectively.","tokens_in":12805,"tokens_out":5949,"duration_ms":57223,"significance":"If the best result were robust, the dual-decoder architecture with code-mixing-specific objectives would be a useful contribution to code-mixed NLP, which currently lacks strong encoder representations for languages like Hinglish. The paper is transparent about its main limitations (small pre-training sample, underparameterized encoder) and ships its tokenizer, code, and data-augmentation prompt, which is a concrete reproducibility asset. The significance is tempered, however, by the absence of error bars, official or published baselines, and a clean ablation isolating the architectural contribution from the objective contribution. The central claim therefore rests on evidence that is internally inconsistent in one case and confounded in another.","major_comments":[{"comment":"The text claims that BiLTM and SPP yield F1 gains of 0.18 and 0.05, respectively, but Table 1 contradicts the ordering. The MLM+BiLTM row reaches F1=0.431; adding SPP to that configuration reduces F1 to 0.295, a drop of 0.136. The 0.046 difference between the MLM+BiLTM+SPP row and the BERTbase row cannot be attributed to SPP because no MLM+SPP condition is reported. Section 9 repeats this unsupported attribution and should be revised or backed by the missing ablation.","section":"§6.2, Table 1"},{"comment":"The claimed BiLTM gain is confounded with the architectural change. Every CMLFormer row uses the dual-decoder architecture with cross-decoder attention, whereas the BERTbase row uses a standard single-stream encoder; no dual-decoder CMLFormer with MLM only is reported. The contribution of BiLTM is therefore not separable from the added decoder parameters and cross-attention mechanism, and an MLM-only CMLFormer ablation is required to support the attribution.","section":"§6.2, Table 1"},{"comment":"The claim of outperforming \"known baselines\" is not established. The comparison model is a BERTbase pre-trained locally on the same 10,000-sentence sample (footnote 2), not the official HingBERT released with L3Cube-HingCorpus, and no published HASOC-2021 results from prior shared-task systems or models such as MuRIL are cited. Without those external comparisons, the headline numbers do not support the phrase \"outperforms known baselines.\"","section":"§3.5.2 and §5"},{"comment":"The results are reported without error bars, repeated seeds, or significance testing. Given the small pre-training corpus (10,000 sentences) and the moderate fine-tuning set (5,740 training examples), deltas such as the accuracy differences of 0.008 and 0.002 in rows 2 and 3 are within ordinary optimization noise. The central performance claim needs at least multiple random seeds with standard deviations or a paired significance test.","section":"§6.2 and Table 1"},{"comment":"The attention analysis is circular as evidence of switching-point sensitivity. The model was explicitly trained to predict the same Gemini-generated switching labels through the SPP objective, so heightened attention at those positions is an expected consequence of supervised training rather than an independent demonstration of emergent code-mixed structure. In addition, the quality of the LLM-generated translations and token-level labels is only manually spot-checked; no quantitative label-quality metric (e.g., agreement with human annotation) is provided, so the auxiliary objectives may propagate label noise.","section":"§6.3 and Appendix A.1.3"}],"minor_comments":[{"comment":"The caption uses \"CLM\" where the text and table header use \"BiLTM\"; this inconsistency should be fixed.","section":"Table 1 caption"},{"comment":"The architecture ablations are described in the future tense (\"we will decouple\", \"we will investigate\") rather than as completed experiments; they should either be actually run or explicitly labeled as planned work.","section":"Appendix A.2"},{"comment":"The CMI weighting factors w_n and w_p and the objective-loss weights α, β, γ, η, ζ, δ are free parameters, but no sensitivity analysis or justification for the chosen values is provided.","section":"Equation (2) and Table 3"},{"comment":"The dataset size is given as 52.93M sentences in footnote 1 and as 52.9M in Section 8; the inconsistency should be corrected.","section":"Footnote 1 and Section 8"},{"comment":"The attention aggregation procedure (layer, head, and normalization) is not specified in enough detail to reproduce Figure 3; the same extraction should be applied to BERTbase and CMLFormer.","section":"Section 6.3, Figure 3"},{"comment":"The appendix ends with a duplicated period (\"switching points..\"), and Section 6.2 contains the typo \"Bilingual Translation Language Modeling\" instead of \"Bilingual Language Translation Modeling.\"","section":"Appendix A.1.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable fit for a code-mixed NLP venue and the authors are transparent about resource constraints, but the empirical evidence in its current form is not sufficient for acceptance. The central problem is that every claimed gain is either contradicted by Table 1 (SPP) or confounded with the architectural change (BiLTM), and the baseline set is too narrow to support the stated conclusions. These issues are fixable with additional experiments, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nShort version: CMLFormer is a real attempt to model code-mixed text with a synchronous dual-decoder Transformer and a set of auxiliary objectives (switching point prediction, bilingual translation modeling, and others). That is genuinely new relative to the cited work. But the paper's headline claims about which objective helps are not supported by its own Table 1.\n\nWhat's good: the architecture is clearly described, the augmentation pipeline (Gemini-generated translations and switching labels) is transparent, and the limitations section is unusually honest. They also ship code and a tokenizer, which is more than many papers do.\n\nThe soft spots are serious. The conclusion credits BiLTM with a 0.18 F1 gain and SPP with 0.05 over BERTbase on HASOC-2021. Table 1 shows something else: row 2 (MLM + BiLTM) gets F1=0.431, and row 3 (adding SPP) drops to 0.295. So SPP as an addition to BiLTM hurts by 0.136, not helps. The 0.05 gain is the difference between BERTbase and the BiLTM+SPP row, but there is no SPP-only condition, so attributing it to SPP is unjustified. Similarly, every CMLFormer row uses the dual-decoder architecture, so the 0.18 \"BiLTM gain\" is confounded with the architectural change. There are no published baselines (they pre-train their own BERTbase on a 10k sample), no error bars, and no repeated runs. The attention analysis is partly circular, since the model is trained on the same switching labels it is shown to attend to.\n\nThat said, the central idea is not silly. Pre-training on translation and switching supervision could plausibly help code-mixed representations. The paper just doesn't demonstrate it cleanly. A proper ablation set would include MLM-only CMLFormer, SPP-only (without BiLTM), and repeated runs with variance. They also need to compare against official HingBERT or other published HASOC results.\n\nWho is this for? People working on code-mixed NLP, especially Hinglish. It's worth a serious referee because the architecture is novel and the objectives are testable, but the current claims need major revision. I'd send it to review with a request for substantial changes, not desk-reject it.\n\nBest,\n[Your name]","headline":"New dual-decoder architecture for code-mixed text, but the paper's own Table 1 undercuts the SPP attribution in the conclusion.","tokens_in":13368,"tokens_out":2489,"would_cite":false,"duration_ms":23623,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dual-decoder Transformer with switching-point and translation pretraining lifts Hinglish hate-speech F1 by 0.18 over a same-data BERT baseline.","keywords":["code-mixed language modeling","switching points","dual-decoder transformer","Hinglish","multitask pretraining","hate speech detection","cross-lingual representation learning","HASOC 2021"],"falsifier":"Retrain CMLFormer on a human-annotated subset of the same 10,000 augmented Hinglish sentences, with manually verified switching-point labels and translations, and compare HASOC-2021 F1 against the model trained on LLM-generated labels; if the human-labeled model does not match or exceed the reported F1 gains, the claimed gains depend on the LLM labeling distribution rather than on switching structure.","tokens_in":12287,"feed_emoji":"🔀","tokens_out":9100,"duration_ms":80632,"temperature":0.7,"pith_summary":"CMLFormer is designed to test a specific hypothesis: a language model pretrained to know where language switches occur in code-mixed text, and to translate those mixed sentences into each constituent language, will represent code-mixed input better than models pretrained only on generic masked-language modeling. The paper builds a Transformer with one shared encoder and two synchronized decoders, one for the base language and one for the mixing language, and trains it on a small augmented Hinglish corpus with switching-point labels and parallel Hindi and English translations. After pretraining, only the encoder is fine-tuned on the HASOC-2021 Hinglish hate-speech benchmark, where the authors report F1 gains of 0.18 and 0.05 over a same-data BERT-base baseline from the translation and switching-point objectives, respectively. If this holds, explicit structural supervision may be a viable route to code-mixed NLP without relying on larger models or larger monolingual corpora.","feed_headline":"Switching-point pretraining lifts Hinglish hate-speech F1 by 0.18","feed_subtitle":"A dual-decoder Transformer taught to spot language transitions beats a same-data BERT baseline on HASOC-2021.","key_machinery":"The load-bearing mechanism is the synchronized dual-decoder cross-attention sub-layer. Two Transformer decoders share one encoder; after each decoder's encoder-decoder attention, the base-language decoder attends to the mixing-language decoder's current hidden states and vice versa, with learned linear projections mapping each decoder's features into the other's space. This couples generation of the Hindi and English translations of the same code-mixed sentence, forcing the encoder to produce representations aligned across both languages. The other mechanism is switching-point prediction: a binary token-level classification head on the encoder, trained with word-level switch labels aligned to subwords, that teaches the encoder where language transitions occur.","core_discovery":"The central discovery claimed is that explicit structural supervision specific to code-mixing—per-token switching-point prediction and bilingual translation into both constituent languages—can teach a shared encoder representations that transfer better to downstream code-mixed classification than masked-language modeling alone. In the reported experiments, the translation objective (BiLTM) added to MLM raises HASOC-2021 hate-speech F1 from 0.249 for a same-data BERT-base to 0.431; the combination of MLM, BiLTM, and switching-point prediction reaches 0.295, which the paper counts as an absolute F1 gain of 0.05 over the baseline. Attention analysis shows the pretrained encoder allocates high attention around switching points regardless of switch direction or position, whereas BERT does not. The paper also reports that adding further objectives (BTSP, TLC, and CMI) degrades downstream performance, a result it attributes to conflicting learning signals and limited encoder capacity rather than to the structural-supervision idea itself.","pith_inferences":["A direct test of the central claim would replace the LLM-generated switching labels with human-annotated labels on a held-out subset and check whether the HASOC F1 gain persists; if it does not, the SPP gain is tied to the label generator rather than to switching structure.","The architecture could transfer to other high-resource code-mixed pairs, but the paper only evaluates Hinglish; a reader should not assume similar gains for pairs with different typology or less Roman-script standardization.","Since the attention analysis is qualitative, a quantitative metric—such as the correlation between attention scores and ground-truth switch positions across layers—could turn the claimed switching sensitivity into a falsifiable intrinsic evaluation.","The observation that BTSP, TLC, and CMI hurt performance suggests the multi-task objective space is not monotone; tuning per-objective weights or using a larger encoder could flip those signs, an experiment the paper explicitly lists as future work."],"forward_implications":["An encoder pretrained with translation and switching-point objectives can be fine-tuned for downstream code-mixed classification at the same inference cost as a standard BERT encoder, since the decoders are detached after pretraining.","Switching-point supervision could be applied to other code-mixed language pairs without parallel corpora, as long as word-level language labels or an LLM annotator can supply switch positions.","The reported degradation from adding BTSP, TLC, and CMI suggests that future multi-task code-mixed pretraining should treat objectives as competing and tune loss weights and encoder size rather than assuming all auxiliary tasks help.","Pretraining on only 10,000 sentences is claimed to be sufficient to produce measurable downstream gains, which would make targeted structural pretraining a cheap alternative to large-scale code-mixed corpus collection."],"supporting_citations":[{"why":"supplies the L3Cube-HingCorpus Hinglish corpus from which the 10,000 pretraining sentences are sampled, along with the HingBERT reference point.","marker":"[Nayak and Joshi, 2022]"},{"why":"defines the HASOC-2021 Hinglish hate-speech benchmark used for fine-tuning and evaluation.","marker":"[Mandl et al., 2021]"},{"why":"provides the BERT architecture, the MLM objective, and the BERT-base baseline the paper compares against.","marker":"[Devlin et al., 2019]"},{"why":"supplies the Transformer encoder-decoder backbone that CMLFormer modifies with dual decoders and cross-attention.","marker":"[Vaswani et al., 2017]"},{"why":"documents the Gemini model used to generate the parallel translations and switching-point labels in the augmented corpus.","marker":"[Anil et al., 2024]"},{"why":"defines the Code-Mixing Index formulation that the CMI regression objective is based on.","marker":"[Das and Gambäck, 2014]"},{"why":"introduces translation language modeling, the cross-lingual pretraining idea that the BiLTM objective builds on.","marker":"[Lample and Conneau, 2019]"}],"fun_headline_variants":["Dual decoder with switch-point pretraining beats BERT on Hinglish","Translation objective lifts code-mixed F1 more than masked LM","CMLFormer learns language switches, improving hate-speech detection","Switching-point attention emerges in code-mixed Transformer","Pretraining on Hinglish translations boosts HASOC F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Gemini-generated switching-point labels and translations are accurate enough that the SPP and BiLTM objectives teach the encoder a genuine structural signal about code-mixing rather than a model of the annotating model's artifacts.","fun_headline_variants_meta":{"raw":{"variants":["Dual decoder with switch-point pretraining beats BERT on Hinglish","Translation objective lifts code-mixed F1 more than masked LM","CMLFormer learns language switches, improving hate-speech detection","Switching-point attention emerges in code-mixed Transformer","Pretraining on Hinglish translations boosts HASOC F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1381,"prompt_tokens":925,"completion_tokens":456,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":383}},"tokens_in":541,"tokens_out":456,"duration_ms":4975,"temperature":1.0,"reasoning_tokens":383,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:30:56.503496+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain CMLFormer on a human-annotated subset of the same 10,000 augmented Hinglish sentences, with manually verified switching-point labels and translations, and compare HASOC-2021 F1 against the model trained on LLM-generated labels; if the human-labeled model does not match or exceed the reported F1 gains, the claimed gains depend on the LLM labeling distribution rather than on switching structure.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the L3Cube-HingCorpus Hinglish corpus from which the 10,000 pretraining sentences are sampled, along with the HingBERT reference point."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the Transformer encoder-decoder backbone that CMLFormer modifies with dual decoders and cross-attention."},{"cited_title":"o rn Gamb \\","cited_arxiv_id":null,"evidence_quote":"defines the Code-Mixing Index formulation that the CMI regression objective is based on."}],"review_version":1}