{"id":"2458b55d-026e-4420-8f2c-5035123afb9a","arxiv_id":"2601.06347","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":9,"one_line_summary":"Multilingual training data and per-language thresholds matter most for universal NER; plain binary cross-entropy and an architecture-matched backbone suffice, yielding OTTER, which outperforms similar-size models by 5.3 F1 and trails a 32B LLM by only 0.2 points when thresholds are tuned per languag","lead":"This paper systematically isolates which design choices matter when building multilingual named-entity-recognition models, testing architectures, backbones, training data, and loss functions one at a time. It then builds OTTER, a compact model that beats similarly sized systems by roughly five F1 points and approaches 27-32B-parameter generative models at a fraction of the compute.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Oracle per-language thresholds selected on test sets inflate the headline 'competitive with 90x larger models' claim; fixed-threshold OTTER is 0.484, not 0.501.","rationale":"The reader's weakest-assumption correctly identifies the per-language test-set threshold protocol as the main soft spot. I agree. The paper's own Limitations passage ('select the threshold that performs best on average across languages') conflicts with Table 6's per-language footnote, so this is not just a matter of conventional evaluation practice; the manuscript is internally inconsistent about the protocol. All four design-choice findings are, as the reader says, well-controlled within the stated setup, and the release of checkpoints strengthens reproducibility. The dataset-comparison and loss-comparison potential confounds (dataset size vs language count; early stopping bias) are real but less directly falsifiable from the text alone; the threshold issue can be settled with one re-evaluation. Since the reader's CONDITIONAL verdict already incorporates this concern, no verdict change is warranted. The concrete test would either confirm the conditional status or force a downgrade to REJECT if the normalized gap grows beyond 2pp without any offsetting evidence.","tokens_in":27915,"tokens_out":8986,"duration_ms":84914,"concrete_test":"Re-run OTTER (cross-enc., mmBERT-100k) on all seven benchmarks, selecting per-language thresholds from held-out development splits (or a single global threshold tuned on training data), and compute the macro-F1 before aggregating. Compare this deployable F1 to the 0.501 and 0.484 rows in Table 6 and to Qwen3-32B's 0.503. If the gap to Qwen3-32B exceeds 2pp, the abstract's 'competitive' claim must be qualified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline comparison in the abstract — 'competitive performance compared to 90x larger generative models' — rests on Table 6's starred row: OTTER (cross-enc.) w/ mmBERT-100k* at 0.501 F1, annotated 'Using the best performing threshold per language before aggregating.' The fixed-threshold row immediately below is 0.484. Thus the 0.2pp gap to Qwen3-32B (0.503) is obtained by choosing a separate decision threshold per language on the same test splits used for the comparison. That is an oracle protocol: in deployment, thresholds would have to be estimated from development data for each language, and the paper does not show those thresholds are stable. The Limitations section says instead that the authors 'select the threshold that performs best on average across languages,' which is a different, aggregate protocol — an internal inconsistency about what was actually reported. If per-language oracle selection overstates the deployable score by even 1.5pp, the claimed 'competitive with 90x larger' narrows to 'within roughly 2-6 F1 points' of Qwen3-32B/Gemma3-27B. The core ablation findings (data diversity, backbone-architecture coupling, BCE suffices) are not directly affected, but the abstract's flagship comparative claim is less secure than it appears.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a controlled empirical study of design choices for universal multilingual NER: (i) bi- vs cross-encoder architectures, (ii) transformer backbone, (iii) training dataset language coverage, and (iv) loss function and threshold selection. It evaluates on seven human-annotated benchmarks covering 250 test splits. The authors report that multilingual training data is the key driver of zero-shot performance, that backbone effectiveness is architecture-dependent when trained on PileNER, and that plain BCE outperforms focal and contrastive losses. They then scale up the best configuration (mmBERT, cross-encoder, BCE, FiNERweb) into OTTER, reporting 0.501 F1 with per-language test-set-optimal thresholds and 0.484 with a fixed threshold, and claim competitiveness with 32B-scale generative models. Appendices document detailed per-backbone/per-dataset results, loss ablations, translated-label experiments, and the effect of word segmentation.","tokens_in":28231,"tokens_out":9407,"duration_ms":88909,"significance":"If the design-choice findings withstand the confounds below, the paper would be a useful reference for practitioners: it isolates components often varied jointly, provides reproducible code and checkpoints, and honestly reports negative results (Shona prefixation, translated-label costs). The paper's main contribution—identifying which levers matter—is valuable, but the headline OTTER comparison and the 'language diversity'/'architecture-dependence' conclusions require verification.","major_comments":[{"comment":"The headline 'competitive performance compared to 90x larger generative models' rests on the starred OTTER row (0.501 F1), which uses 'the best performing threshold per language before aggregating'—i.e., test-set-optimal thresholds. The fixed-threshold variant scores 0.484, a ~2pp gap to Qwen3-32B (0.503), not 0.2pp. Selecting thresholds on the evaluation test set is an oracle protocol and does not reflect deployable performance. Moreover, the Limitations say the authors 'select the threshold that performs best on average across languages,' which is a different, aggregate protocol, creating an internal inconsistency. Please report a development-set threshold-selection protocol, or clearly label the 0.501 as an upper bound and base the abstract claim on the non-oracle number.","section":"Section 4 / Table 6 / Limitations (Threshold Selection)"},{"comment":"The conclusion that 'a training dataset covering more languages is key to good performance' is confounded: PileNER, Euro-GLiNER-x, and FiNERweb differ in total size, label distribution, annotation pipeline, and domain, not only in language count. The gains of up to +10 F1 (Table 4) cannot be attributed to language coverage without a size-matched control. The subword-gradient statistic in Table 3 is suggestive but not sufficient. Add an equal-size comparison (e.g., subsample the multilingual corpora) or explicitly soften the causal claim.","section":"Section 3.2 / Table 4 / Conclusion (i)"},{"comment":"The claim that optimal backbone choice is architecture-dependent is contradicted by the paper's own later results. Table 2 (PileNER, early stopping) shows RemBERT best for bi-encoder and XLM-R best for cross-encoder; Table 4 (FiNERweb) shows mmBERT best for both architectures (0.437 bi, 0.461 cross) with RemBERT second for both. The architecture-dependence finding therefore does not generalize to the final training setup, yet the conclusion and abstract retain it. Please qualify the claim or reconcile the evidence.","section":"Section 3.1 vs. Section 3.2 / Conclusion (ii)"}],"minor_comments":[{"comment":"'a simply using a binary cross-entropy loss' should be 'simply using'.","section":"Abstract and Introduction"},{"comment":"'closing the gap to 0.3pp F1' mismatches Table 6, where Qwen3-32B is 0.503 and OTTER* is 0.501—a 0.2pp gap.","section":"Section 4"},{"comment":"Duplicate citation: 'mT5-base (Xue et al., 2021) (Xue et al., 2021)'.","section":"Section 3.1"},{"comment":"'Masakhaner' capitalization is inconsistent with 'MasakhaNER' used elsewhere.","section":"Table 1"},{"comment":"Dataset name spelled inconsistently as 'FiNERWeb' and 'FiNERweb'; please standardize.","section":"Throughout"},{"comment":"Sentence 'we use standard binary cross-entropy loss and leave the exploration of different datasets and loss functions to later sections' is ungrammatical; please revise.","section":"Section 2.1"}],"recommendation":"major_revision","confidential_remarks":"The FiNERweb dataset (Golde et al., 2025) is an overlapping-author artifact central to the training-data finding. This is not itself a flaw, but the paper should make the controlled comparison explicit; as written, the language-coverage conclusion is at risk of size confound. The threshold protocol is the most important fix before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Key finding: this is the first controlled comparison that varies architecture, backbone, dataset, and loss one at a time for universal multilingual NER. That alone is worth having. The paper does it cleanly, with seven external human-annotated benchmarks, full threshold sweeps in the appendix, and honest negative findings — the Shona prefixation failure and the translated-label cost. The main takeaways are plausible and useful: language coverage in the training corpus matters most; plain BCE beats focal and contrastive losses everywhere tested; and backbone choice is architecture-dependent, with RemBERT at 0.379 F1 as a bi-encoder collapsing to 0.206 as a cross-encoder.\n\nThe soft spot is the headline comparison to '90x larger generative models.' That claim uses the starred row in Table 6: OTTER cross-encoder with per-language thresholds selected on the test splits (0.501 F1), not the fixed-threshold 0.484. Choosing the best threshold per language after seeing the test labels is an oracle protocol, and the paper itself is inconsistent about it — the Limitations say they select the threshold that performs best on average across languages, which is different. Unless the authors show per-language thresholds estimated from development data transfer to the test splits, the honest summary is 'within 2-6 F1 points of 27-32B LLMs,' not '0.2pp gap.' That does not sink the ablation findings, which are robust to this issue, but it changes the abstract's bragging rights.\n\nTwo smaller issues: no seed or variance information anywhere, so we don't know how stable the 1-2pp differences are; and the no-word-segmentation design systematically underperforms on Shona due to prefixation, which is a real limit on the 'universal' claim even though the authors disclose it.\n\nWho this is for: anyone building or benchmarking multilingual NER systems. The paper deserves a serious referee. It would come back with a request to fix the threshold protocol and add variance, not with a rejection.","headline":"A solid controlled ablation of multilingual NER design choices; the headline LLM-comparison rests on test-set-tuned per-language thresholds, but the core findings stand.","tokens_in":28776,"tokens_out":1418,"would_cite":true,"duration_ms":13570,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper isolates which design choices actually drive universal multilingual named entity recognition, arguing that language coverage in training data is the dominant lever, backbone choice depends on architecture, and plain binary cross-","keywords":["named entity recognition","multilingual NER","design choices ablation","binary cross-entropy loss","cross-encoder vs bi-encoder","multilingual training data","threshold selection","zero-shot cross-lingual transfer"],"falsifier":"Evaluate the released OTTER checkpoint using a single decision threshold selected only on held-out validation data (no per-language test-set tuning), and compare against the reported per-language-oracle average: if the fixed-threshold F1 drops by more than the reported ~1.5 points, or if the gap to the 32B generative model widens to more than ~6 points, the headline parity claim is not reproducible. A second check: train the identical recipe on English-only data and compare on languages absent from both corpora—no gain would refute the language-coverage claim.","tokens_in":27708,"feed_emoji":"🌐","tokens_out":6995,"duration_ms":68300,"temperature":0.7,"pith_summary":"The paper sets out to end the practice of bundling design decisions in multilingual NER research. It runs controlled ablations over four dimensions—architecture, backbone, training data, and loss—and claims that three levers matter most: the number of languages in the fine-tuning corpus, matching the backbone to the encoder architecture, and per-language decision thresholds. The loss function, by contrast, is nearly irrelevant: plain binary cross-entropy beats focal and contrastive losses in every comparison. Combining those findings into a single model (OTTER) trained on 91-language data supports over 100 languages and, the authors report, outperforms similarly sized models by 5.3 F1 points while staying within a few points of generative models roughly 90 times larger. A sympathetic reader would take away a practical recipe: spend your budget on multilingual data and threshold calibration before trying clever objectives.","feed_headline":"Language coverage, not loss tricks, drives multilingual NER","feed_subtitle":"A 100-language model using plain binary cross-entropy rivals generative models 90 times its size.","key_machinery":"The argument is carried by a span-scoring architecture that can run in two modes: a cross-encoder, which concatenates text and label descriptions into one transformer, and a bi-encoder, which encodes text and labels separately and compares them afterward. Both produce span representations (concatenating projected start, end, and span-width embeddings) scored against label representations; a binary cross-entropy loss over every span–label pair treats the severe class imbalance of negative spans as a thresholding problem. Around this shared core, the paper varies one design dimension at a time under fixed hyperparameters, which is the mechanism that lets it attribute performance differences to","core_discovery":"In the paper's own terms: when architecture (cross-encoder vs bi-encoder), transformer backbone, fine-tuning dataset, and loss function are evaluated in isolation rather than in combination, performance is driven by (i) the language coverage of the training corpus—switching from English-only PileNER to 91-language FiNERweb raises F1 by up to 10 points and updates more than 94% of subword embeddings versus 38.5%; (ii) an architecture-backbone match—RemBERT is the best bi-encoder at 0.379 F1 but collapses to 0.206 F1 as a cross-encoder, while other backbones behave in the opposite way; and (iii) decision-threshold selection, where per-language thresholds add about 1.5 F1 points. Binary cross-e","pith_inferences":["If language coverage is the dominant lever, one testable prediction is a smooth scaling curve of F1 with the number of training languages; the paper's three-point comparison (1, 12, 91 languages) is consistent with but does not yet prove such a curve.","The per-language threshold effect is plausibly driven by tokenizer and script differences shifting score distributions; if so, thresholds might be predicted from tokenizer statistics alone, removing the need for labeled validation data.","The documented failure on languages with productive prefixation (e.g., Shona) suggests a concrete fix—augmenting multilingual training data with locative/associative prefix constructions—that the paper identifies but does not pursue.","The loss-function irrelevance may extend beyond NER: if span-label scoring is the bottleneck, label-text encoding capacity and data coverage should dominate objective engineering in other span-based extraction tasks as well."],"forward_implications":["Training on more languages in the fine-tuning corpus is the single largest controllable gain; adding coverage updates far more subword embeddings and improves zero-shot transfer across all tested backbones and architectures.","Backbone rankings do not transfer across architectures: a strong bi-encoder backbone can be a weak cross-encoder, so backbone choice must be re-tested when the architecture changes.","A plain binary cross-entropy loss is sufficient; upweighting positives, focal loss, and contrastive loss do not improve over it and can slow convergence.","Per-language decision thresholds are worth about 1.5 F1 points and narrow the gap to much larger generative models; threshold behavior is more stable for bi-encoders than cross-encoders.","A task-specific encoder model trained on multilingual synthetic data can approach the multilingual NER quality of generative models tens of times larger, at a fraction of inference cost."],"fun_headline_variants":["Coverage beats loss tricks in multilingual NER","Why language coverage is key for multilingual NER","Multilingual NER: Architecture-backbone match matters","Data coverage drives multilingual NER, not loss functions","Otter: Efficient multilingual NER via coverage and thresholds"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The strongest headline claim assumes that per-language thresholds tuned on the evaluation test sets reflect deployable performance; with a single fixed threshold, the reported average drops by about 1.5 points, so 'competitive with 90x larger generative models' would narrow to 'within a few points' in real-world use.","fun_headline_variants_meta":{"raw":{"variants":["Coverage beats loss tricks in multilingual NER","Why language coverage is key for multilingual NER","Multilingual NER: Architecture-backbone match matters","Data coverage drives multilingual NER, not loss functions","Otter: Efficient multilingual NER via coverage and thresholds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"total_tokens":1389,"prompt_tokens":726,"completion_tokens":663,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":470,"completion_tokens_details":{"reasoning_tokens":588}},"tokens_in":470,"tokens_out":663,"duration_ms":6809,"temperature":1.0,"reasoning_tokens":588,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T11:26:35.068854+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate the released OTTER checkpoint using a single decision threshold selected only on held-out validation data (no per-language test-set tuning), and compare against the reported per-language-oracle average: if the fixed-threshold F1 drops by more than the reported ~1.5 points, or if the gap to the 32B generative model widens to more than ~6 points, the headline parity claim is not reproducible. A second check: train the identical recipe on English-only data and compare on languages absent from both corpora—no gain would refute the language-coverage claim.","supporting_citations":[],"review_version":1}