{"id":"7bc7b7da-6cb5-4b3e-b91c-24a4f5e1f43f","arxiv_id":"2509.02514","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Fine-tuned code-mixed models (HingBERT, HingMBERT, HingRoBERTa) reach higher F1 than multilingual BERTs and zero-shot Google Gemini on Hinglish named entity recognition.","lead":"This paper compares how well different language models find names of people, organizations, and places in Hinglish text, a mix of Hindi and English. It finds that models trained on code-mixed data score higher than a large general chatbot, Google Gemini, used with no training examples.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No error bars or significance tests support the 2.24-point HingBERT-over-MuRIL F1 gap; validation/test rankings flip, and the authors' own limitations concede the small dataset, so the attribution to code-mixed pretraining is not established.","rationale":"The reader's weakest assumption—that the 2-point F1 differences are meaningful without error bars—is exactly the load-bearing point. I agree. The paper's central claim is a comparative ranking, and the only evidence is point estimates in Tables 2 and 3. The instability of the code-mixed model rankings between validation and test (HingMBERT first on validation, HingBERT first on test) internal to the paper demonstrates that the test-set F1 for similar models varies by at least ~1.4 points. Therefore the 2.24-point gap between HingBERT and MuRIL could easily be due to the same randomness. The rare I-Org and I-Loc tags are a secondary issue; seqeval's overall F1 is micro-averaged over spans, so those tags have limited impact, but the small test set overall remains a concern. I also noted an ambiguity in Section 3.4: the paper says models are 'retrained on the full training data' after Optuna, but Section 4.2 applies early stopping on a validation F1, and it is never explained how the validation split is obtained from the 80/20 partition. This could be a reproducibility failure, but I did not make it the primary attack because it is more about missing detail than an identified error. The primary concern is the absence of any statistical support for the claimed ordering. The reader's CONDITIONAL verdict is appropriate; the authors should be required to provide uncertainty estimates and code. Therefore verdict_should_be is UNCHANGED.","tokens_in":8395,"tokens_out":9875,"duration_ms":82619,"concrete_test":"Require the authors to release the per-seed test predictions or rerun the evaluation on 10 random 80/20 splits, and compute the bootstrap 95% confidence interval for the HingBERT minus MuRIL F1 difference. If the interval includes zero, or the difference is not positive in the large majority of splits, the claimed advantage of code-mixed pretraining is not established by this study.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 defines an 80/20 split of 3,637 sentences; Table 1 shows only 90 I-Org and 31 I-Loc instances in the entire corpus, so the test set contains roughly 18 and 6 of these rare tags. Section 4.2 states each experiment was repeated with three seeds and the average reported, but no standard deviations or confidence intervals are given anywhere. Table 3 reports test F1 of 79.74 for HingBERT and 77.50 for MuRIL, a 2.24-point gap. The total number of entity spans in the test set is on the order of 866 (20% of B-Per+B-Org+B-Loc), so this gap corresponds to about 19 entity spans. Without variance, 19 spans out of ~866 is not clearly beyond noise. Moreover, the within-code-mixed rankings flip between validation and test: HingMBERT is first on validation (77.14, Table 2) but third among code-mixed models on test (78.87, Table 3), while HingBERT moves from third (76.51) to first (79.74). This spread (~1.4 points) is comparable to the 2.24-point gap to MuRIL, suggesting the test-set differences are within the noise of the evaluation. The Limitations section explicitly concedes the 'relatively small dataset size.' Consequently, the paper's attribution of the gap to domain-specific pretraining is unsupported without an uncertainty analysis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a comparative evaluation of three families of models for Hindi-English code-mixed named entity recognition: code-mixed fine-tuned BERT variants (HingBERT, HingMBERT, HingRoBERTa), non-code-mixed multilingual models (BERT Base Cased, IndicBERT, RoBERTa, MuRIL), and a zero-shot generative LLM (Google Gemini). All models are evaluated on a 3,637-sentence Hinglish NER dataset using an 80/20 train/test split, with entity-level precision, recall, and F1 computed via seqeval. The reported test F1 scores place HingBERT first at 79.74, followed by HingMBERT (78.87), HingRoBERTa (78.39), and the best non-code-mixed model MuRIL (77.50), with Gemini at 62.24. The paper attributes this gap to domain-specific code-mixed pretraining and argues that specialized models still outperform zero-shot LLMs.","tokens_in":8712,"tokens_out":4434,"duration_ms":40866,"significance":"If the reported ranking is statistically robust, the paper provides a useful empirical reference for code-mixed NER, confirming the practical value of domain-specific pretraining over generic multilingual models and quantifying the current zero-shot performance ceiling of a commercial LLM. The use of Optuna-tuned hyperparameters for each model and the entity-level evaluation protocol are commendable. However, the significance is tempered by several issues: there is no uncertainty quantification despite the claim of three-seed averaging, the test-set differences are small in absolute terms, the dataset provenance is not clearly identified, and the paper itself concedes the dataset is relatively small. These limitations directly affect the strength of the central attribution claim.","major_comments":[{"comment":"The central comparative claim rests on gaps measured in entity-level F1 points (HingBERT 79.74 vs. MuRIL 77.50 on the test set, Table 3), but the paper reports only averages from three seeds with no standard deviations, confidence intervals, or significance tests anywhere. Based on the label counts in Table 1 and the 80/20 split, the test set contains on the order of 866 entity spans; a 2.24-point gap corresponds to roughly 19 spans, which is not clearly beyond seed-level or data-sampling noise. This concern is reinforced by the validation/test ranking reversal: HingMBERT is first on validation (77.14, Table 2) but third among code-mixed models on test (78.87, Table 3), while HingBERT moves from third on validation (76.51) to first on test (79.74). The paper's Limitations section itself acknowledges the 'relatively small dataset size.' I request that the authors report per-seed results, standard deviations, and a statistical significance test (e.g., a bootstrap over test sentences or a paired test) before the claim that code-mixed pretraining explains the gap can be accepted.","section":"§4.2, Tables 2–3, Limitations"},{"comment":"The dataset is described only as 'a benchmark Hinglish NER dataset,' but its source is never explicitly identified or cited. This is a reproducibility issue: the reader cannot verify the split, compare with prior results on the same benchmark, or assess the claim that this is a standard evaluation setting. The authors should state whether this is the Singh et al. (2018) corpus or another dataset, describe how the 80/20 split was generated (random seed, stratification), and document any preprocessing applied to the original corpus. Additionally, Table 1 shows that the full corpus contains only 90 I-Org and 31 I-Loc instances; the test set therefore contains very few examples of these rare tags. The paper should report per-entity F1 with confidence intervals for these categories, as the current single macro/entity-level F1 may be dominated by frequent types.","section":"§3.1, Table 1"},{"comment":"The zero-shot Gemini evaluation is not reproducible as described. The paper does not provide the prompt template, the exact model version or access date, decoding parameters (temperature, max tokens), or the post-processing and alignment procedure used to convert Gemini's free-form output into BIO tags aligned with the original token boundaries. Since the Gemini result (F1 62.24) is a reported contribution and is compared against fine-tuned models, these details are essential. I request the authors include the full prompt, the parsing pipeline, and any failure cases where the model output could not be aligned.","section":"§3.6, §4.3"}],"minor_comments":[{"comment":"The text says 'Among them, HingMBERT achieves the highest F1-score' without specifying the evaluation set; this is true only on the validation set, while on the test set HingBERT is highest. Please clarify.","section":"§5, Tables 2–3"},{"comment":"The model name is spelled inconsistently as 'Hing-mBERT' in Section 3.3 and 'HingMBERT' in the tables; please unify the nomenclature.","section":"§3.3, Tables 2–3"},{"comment":"The paper mentions 'macro-averaged precision and F1-score' in the results discussion, but the tables do not state the averaging scheme used by seqeval; please specify whether the reported F1 is a macro-average over entity types and whether 'Accuracy' is token-level accuracy with masked subwords.","section":"§3.5, §5"},{"comment":"The numerical formatting is inconsistent (e.g., 96.88 vs. 97.38 with different decimal precision); please use a consistent number of decimal places throughout.","section":"Tables 2 and 3"},{"comment":"The relationship between the validation set used for Optuna optimization and early stopping and the 80/20 split is not fully specified; please clarify whether the validation set is a subset of the training portion or a separate split, and report the validation-set construction method.","section":"§3.4, §4.2"}],"recommendation":"major_revision","confidential_remarks":"The paper relies heavily on prior work from the same research group (L3Cube Labs / IIT Madras) for the code-mixed models and for the comparative framing. This is not by itself problematic, but an independent evaluation dataset or external baseline results would strengthen the paper's objectivity. The main issue is the lack of uncertainty quantification, which is fixable. There is also a question of scope: this is a relatively straightforward empirical comparison with a single dataset and a single LLM; the contribution is modest. If the journal targets high-impact NLP venues, the authors may need to add more analysis (per-entity breakdown, error analysis, or cross-dataset validation) to justify publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a serviceable benchmark comparison, not a new mechanism. It confirms what the same group already reported in Patil et al. (2023) and Nayak and Joshi (2022): HingBERT-family models beat generic multilingual BERTs on Hinglish NER. The genuinely new pieces are the zero-shot Gemini evaluation (62.2 F1, far below the fine-tuned models) and the inclusion of IndicBERT, MuRIL, HingMBERT, and HingRoBERTa on this dataset. The paper is clearly written, uses entity-level seqeval metrics, and does a reasonable Optuna hyperparameter search. The authors also state their limitations explicitly, including the small dataset.\n\nThe soft spot is the missing uncertainty analysis. Section 4.2 says each run used three seeds and the average was reported, but no standard deviations or confidence intervals appear anywhere. The test F1 gap between HingBERT (79.74) and MuRIL (77.50) is 2.24 points; with roughly 866 entity spans in the test set, that is around 19 spans. Without variance, we cannot tell whether that gap is signal or noise. The validation table makes this worse: among the code-mixed models, HingMBERT is first on validation and third on test, while HingBERT moves from third to first. The spread among these models is about 1.4 points, similar to the gap to MuRIL, so the exact ordering is unstable. The dataset also has very few rare tags (I-Org: 90, I-Loc: 31 in the whole corpus), so the test set probably contains only a handful of those, making per-type conclusions shaky. No code, no data split, no Gemini prompt or post-processing details, and the dataset is not named; reproducibility is limited.\n\nThe broad conclusion that code-mixed models as a group beat generic multilingual models is probably right—it is consistent with prior work and the direction is stable. But attributing the gap to domain-specific pretraining is an interpretation, not a demonstrated effect. The comparison is fine-tuning on the same training set from different starting points; the authors need to show the gap is robust.\n\nWho this is for: practitioners working on Hinglish NER who want a quick model comparison and a Gemini zero-shot datapoint. It does not change methodology. If I were an editor, I would send it to review—the work is honest and the Gemini result is worth putting on record—but I would require revisions that add error bars or significance testing, release code and the exact split, and soften the causal claim. A referee should not desk-reject it, but should not accept it as is.","headline":"Useful benchmark snapshot with a new Gemini zero-shot datapoint, but the headline ranking lacks error bars and is likely within noise; worth reviewing with required revisions.","tokens_in":9282,"tokens_out":4191,"would_cite":false,"duration_ms":34279,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuned code-mixed BERT models beat zero-shot LLMs on Hinglish NER.","keywords":["code-mixed NER","Hinglish","BERT","named entity recognition","zero-shot LLM","domain-specific pretraining","multilingual models","Hindi-English"],"falsifier":"Run a bootstrap test over the 3,637 sentences with repeated 80/20 splits: if the 95% confidence intervals for entity-level F1 of HingBERT (79.74) and MuRIL (77.50) overlap, the claimed advantage of code-mixed pretraining over MuRIL is not significant at this dataset size.","tokens_in":8207,"feed_emoji":"🏷️","tokens_out":8720,"duration_ms":70171,"temperature":0.7,"pith_summary":"This paper sets out to show that for named entity recognition (NER) in Hindi-English code-mixed text, models fine-tuned on code-mixed data beat both general multilingual transformers and zero-shot large language models. It compares three Hinglish-tuned BERT variants (HingBERT, HingMBERT, HingRoBERTa) against four non-code-mixed baselines (BERT, RoBERTa, IndicBERT, MuRIL) and Gemini on a 3,637-sentence benchmark. The best model, HingBERT, reaches an entity-level F1 of 79.74, while the strongest non-code-mixed baseline MuRIL reaches 77.50 and zero-shot Gemini reaches 62.24. The authors attribute the gap to domain-specific pretraining on real code-mixed corpora, arguing that general multilingual pretraining transfers only partially to informal, transliterated, code-switched text. A reader should care because code-mixed content is pervasive in multilingual societies, and the result clarifies when specialized models are still worth training rather than relying on large general-purpose LLMs.","feed_headline":"Fine-tuned Hinglish BERT beats zero-shot Gemini by 17 F1 points","feed_subtitle":"On a Hinglish NER test set, specialized code-mixed models score 79.7 while zero-shot Gemini scores 62.2.","key_machinery":"The argument is carried by the comparison of three families of models on one benchmark: code-mixed fine-tuned transformers (HingBERT, HingMBERT, HingRoBERTa), non-code-mixed multilingual transformers (BERT Base Cased, RoBERTa, IndicBERT, MuRIL), and a zero-shot LLM (Gemini). The load-bearing mechanism is domain-specific pretraining and fine-tuning on Hindi-English code-mixed corpora, which exposes the model to transliteration, informal grammar, and language switching that standard multilingual corpora lack. All models are fine-tuned using a hyperparameter search and evaluated with entity-level F1 rather than token-level accuracy, so the reported gap is specifically about the ability to recognize complete named entities rather than individual tokens.","core_discovery":"The central discovery is that code-mixed fine-tuned transformer models consistently outperform both non-code-mixed multilingual models and a zero-shot generative LLM on Hinglish NER. On the held-out 20% test set, HingBERT achieves entity-level F1 79.74, HingMBERT 78.87, and HingRoBERTa 78.39, compared with 77.50 for MuRIL, 76.47 for IndicBERT, 76.24 for RoBERTa, and 72.12 for BERT Base Cased. Gemini, evaluated zero-shot with entity labels removed, reaches 62.24 F1. The authors interpret this as evidence that domain-specific pretraining on code-mixed data, rather than model scale or general multilingual coverage, is the decisive factor for structured prediction in code-mixed text. They also note that Gemini's competitive accuracy but much lower precision and recall indicates LLMs still struggle with fine-grained entity classification without supervision.","pith_inferences":["Inference: Because the dataset has only 90 I-Org and 31 I-Loc labels in total, the reported entity-level F1 is driven almost entirely by person entities and beginning tags; a per-tag breakdown could reveal that code-mixed pretraining's advantage does not extend to rare inside-organization or inside-location spans.","Inference: The paper reports no variance estimates, so the two-point gap between HingBERT and MuRIL could be within run-to-run noise; a bootstrap resampling of the 3,637 sentences with repeated splits would test whether the ranking is stable.","Inference: The zero-shot setup uses a single prompt style; testing Gemini with few-shot exemplars or constrained output decoding might close much of the 17-point gap and would clarify whether the deficit is inherent to LLMs or an artifact of the prompting strategy.","Inference: If the pattern transfers, other code-mixed pairs such as Marathi-English will likely see larger NER gains from building a small code-mixed pretraining corpus than from adopting a larger general-purpose multilingual model."],"forward_implications":["Teams building NER for Hindi-English social media text should fine-tune code-mixed BERT variants rather than rely on generic multilingual transformers.","Zero-shot LLMs are not yet a drop-in replacement for task-specific models on structured prediction in code-mixed text, at least without additional prompting or adaptation.","Domain-specific pretraining on real code-mixed data yields a larger benefit than switching to a more powerful general model for this task.","The two-point edge of HingBERT over MuRIL suggests the advantage of code-mixed pretraining is real but modest, and larger benchmarks are needed to confirm its size."],"supporting_citations":[{"why":"Supplies the Hindi-English code-mixed social media NER dataset that the study splits 80/20 for training and testing.","marker":"(Singh et al., 2018a)"},{"why":"Provides HingCorpus and the HingBERT/HingRoBERTa models, the code-mixed pretrained systems whose superiority is the paper's central claim.","marker":"(Nayak and Joshi, 2022)"},{"why":"Introduces Gemini, the zero-shot large language model whose 62.24 F1 serves as the generative baseline.","marker":"(Team et al., 2023)"},{"why":"Earlier comparative study reporting that HingBERT outperforms standard BERT, RoBERTa, and mBERT on Hindi-English tasks, the expectation this paper re-examines for NER.","marker":"(Patil et al., 2023)"},{"why":"Provides the GLUECoS code-switched evaluation benchmark and the evaluation context in which code-mixed models like HingBERT were previously validated.","marker":"(Khanuja et al., 2020)"}],"fun_headline_variants":["Specialized BERT tops zero-shot Gemini by 17 F1 on Hinglish NER","Domain-specific pretraining beats LLM scale for code-mixed NER","Fine-tuned BERT beats zero-shot Gemini on Hinglish by 17 F1","Hinglish NER: fine-tuned models outperform Gemini by 17 F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central ranking assumes that F1 differences of around two points on a single 80/20 split of 3,637 sentences are meaningful, even though the test set has very few examples of rare entity types and no confidence intervals are reported.","fun_headline_variants_meta":{"raw":{"variants":["Specialized BERT tops zero-shot Gemini by 17 F1 on Hinglish NER","Domain-specific pretraining beats LLM scale for code-mixed NER","Fine-tuned BERT beats zero-shot Gemini on Hinglish by 17 F1","Hinglish NER: fine-tuned models outperform Gemini by 17 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001327,"raw_usage":{"total_tokens":5431,"prompt_tokens":1005,"completion_tokens":4426,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":4335}},"tokens_in":621,"tokens_out":4426,"duration_ms":30508,"temperature":1.0,"reasoning_tokens":4335,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:37:23.183030+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a bootstrap test over the 3,637 sentences with repeated 80/20 splits: if the 95% confidence intervals for entity-level F1 of HingBERT (79.74) and MuRIL (77.50) overlap, the claimed advantage of code-mixed pretraining over MuRIL is not significant at this dataset size.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides HingCorpus and the HingBERT/HingRoBERTa models, the code-mixed pretrained systems whose superiority is the paper's central claim."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GLUECoS code-switched evaluation benchmark and the evaluation context in which code-mixed models like HingBERT were previously validated."}],"review_version":2}