{"id":"83395813-fa7f-4060-be51-ae6d9c295741","arxiv_id":"2502.06692","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SLIDE introduces a manually labeled multi-label evaluation set and BERT/FastText models for Scandinavian language identification, using machine translation identity as a silver-labeling signal.","lead":"The authors release SLIDE, a manually annotated multi-label dataset for identifying Danish, Norwegian Bokmål, Norwegian Nynorsk, and Swedish in short texts, plus a family of models. They also show that machine translation can cheaply generate multi-label training data by treating unchanged translations as evidence that a sentence is valid in the target language.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim rests on manual test set with no reported inter-annotator agreement; Limitations concedes Norwegian annotators' lower productive skills in Danish/Swedish may cause mislabeling, so the 5% multi-label rate is unverified.","rationale":"We agree with the reader's CONDITIONAL verdict and with the concern about the MT identity heuristic in Section 3.3; that heuristic is indeed unvalidated, and a training run without silver labels would be informative. However, for the central necessity claim, the more load-bearing assumption is the quality of the manual gold labels. The 5% multi-label figure is the quantitative basis for 'necessary for any accurate LID method,' and it is produced by a small team of Norwegian annotators without agreement statistics. The paper itself flags mislabeling risk in its Limitations section. Additionally, exact-match accuracy structurally prevents any single-label model from scoring on multi-label gold, so the threshold for 'necessary' is set by the annotation conventions, including proper names and punctuation. Re-annotation by native speakers of Danish and Swedish, with and without the contested conventions, would directly test whether the multi-label rate and model rankings are stable. We therefore recommend keeping the CONDITIONAL verdict; the concern does not move the verdict, but it sharpens the required revisions.","tokens_in":12689,"tokens_out":10743,"duration_ms":100688,"concrete_test":"Draw a stratified sample of the SLIDE test set, oversampling all multi-label instances and all sentences shorter than six words. Have three independent annotators per item—one native Danish speaker, one native Swedish speaker, and one native Norwegian speaker—apply the Section 3.2 guidelines, with a separate sensitivity pass that does not treat proper-name-only strings as belonging to all languages. Compute per-label Cohen's kappa and Krippendorff's alpha, and recompute the multi-label rate under majority vote. Then re-run Table 4's exact-match evaluation on the majority-vote labels. If agreement is below 0.8 or the majority-vote multi-label rate falls below the reported 5%, the paper must temper the necessity claim and re-report headline numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The abstract's central claim—'the ability to identify multiple languages simultaneously is necessary for any accurate LID method'—is operationalized as exact-match accuracy on the newly created SLIDE test set. That test set is the sole empirical evidence for the necessity claim, yet its reliability is not demonstrated. Section 3.2 reports that all annotators were native or near-native Norwegian speakers; no inter-annotator agreement scores are given. The Limitations section explicitly concedes: 'while all Norwegians generally understand Swedish and Danish well... their productive capabilities are much lower, and there might be cases of mislabeling.' The annotation guidelines also encode contestable conventions: proper-name-only instances are labeled with all four Scandinavian languages, and Danish/Bokmål sentences differing only in optional comma placement are deliberately labeled as both. These decisions inflate the reported 5% multi-label rate and directly penalize single-label baselines under exact-match. If a substantial fraction of the multi-label gold instances are annotation artifacts, the necessity claim is weakened even if the released models are useful. The absence of native Danish/Swedish annotator validation or agreement statistics is therefore the most load-bearing unmeasured assumption.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SLIDE, a manually annotated multi-label evaluation dataset for sentence-level language identification among Danish, Norwegian Bokmål, Nynorsk, and Swedish, built from Universal Dependencies treebank splits. The authors also release a family of LID models—three NorBERT3-based transformers and a fast static-embedding model—trained with a proposed silver-labeling method that uses machine translation identity: if a sentence is unchanged by translation into a target language, it is labeled as valid in that language. On the SLIDE test set, the best model reaches 96.4% exact-match accuracy, outperforming baselines such as GlotLID (93.4%), and the paper claims that multi-label output is necessary for any accurate LID method. The authors additionally report out-of-domain results on the Haas and Derczynski (2021) test sets, where SLIDE-base is slightly below GlotLID.","tokens_in":12915,"tokens_out":6045,"duration_ms":53594,"significance":"The manually curated multi-label dataset is a useful resource, and the released models with reported speed-accuracy trade-offs are a practical contribution. The evaluation is not circular: the test set labels are manually annotated and independent of the training pipeline, and the MT-identity heuristic produces training labels rather than model-derived predictions. However, the central abstract claim is considerably stronger than the evidence: it rests on one test set, the annotation reliability of that test set is not demonstrated, the MT silver-labeling heuristic is never validated, and the out-of-domain results do not support the word 'any.' With targeted additional analyses and a more careful claim, the contributions would be publishable; as written, the central claim overreaches.","major_comments":[{"comment":"The claim that multi-label output is 'necessary for any accurate LID method' is not supported by the reported experiments. Table 4 shows a three-point exact-match gap between SLIDE-base (96.4) and GlotLID (93.4) on a single 6,950-sentence test set, but exact-match accuracy by construction gives no credit to single-label classifiers on the 5% of gold instances that are multi-label, so the gap conflates multi-label handling with overall classifier quality. To make the claim load-bearing, please report performance on the single-label subset, give an upper bound for any single-label classifier on the multi-label instances (for example, loose accuracy restricted to those instances), and quantify how much of the exact-match gap is actually attributable to multi-label gold instances rather than ordinary misclassification. Absent that analysis, the claim should be narrowed to 'on the SLIDE test set, exact-match accuracy requires multi-label predictions.'","section":"Abstract and §6.2"},{"comment":"The MT-identity silver-labeling heuristic is never validated. The method assumes that NorMistral is conservative and that an unchanged translation is a reliable positive signal for the target language, but no evidence is provided for this assumption. Since the multi-label training signal and the claimed 'novel approach' both depend on this heuristic, please validate it directly, for example by comparing MT-derived multi-labels against the manually annotated SLIDE validation/test instances, or by manual inspection of a sample of identity translations. Without such validation, the reported multi-label model performance could reflect noise from erroneous silver labels rather than genuine multi-label supervision.","section":"§3.3"},{"comment":"The manual test set, which is the sole empirical basis for the central claim, lacks inter-annotator agreement statistics, and the Limitations section concedes that the Norwegian annotators' productive skills in Danish and Swedish are lower and that mislabeling may occur. The annotation conventions for proper-name-only sentences and for optional comma placement also directly increase the multi-label rate: proper-name-only instances are labeled with all four Scandinavian languages, and Danish/Bokmål sentences differing only by a comma are labeled as both. Please report agreement scores, provide a breakdown of multi-label instances by category, and run a sensitivity analysis excluding or re-labeling the contested categories. If the 5% multi-label rate changes substantially under that analysis, the necessity claim is weakened.","section":"§3.2 and Limitations"},{"comment":"The out-of-domain results weaken the generality of the 'any accurate LID method' claim. On the Haas and Derczynski (2021) test sets, SLIDE-base achieves 92.7 and 95.3 accuracy, slightly below GlotLID's 93.0 and 95.7. The authors attribute this to preprocessing and mislabeled sentences, but those are exactly the conditions under which a generally accurate LID method must operate. Either temper the abstract claim to the SLIDE test set or present additional evidence, such as results on other out-of-domain data, that SLIDE retains its advantage beyond the treebank-derived evaluation set.","section":"§6.3"}],"minor_comments":[{"comment":"The sentence 'We also find 29444 examples of a comma preceding at 'that' in the Norwegian LBK corpus' appears to contain a typo and unclear phrasing; it should read something like '29,444 examples of a comma preceding 'at'' or 'a comma before the conjunction 'that''.","section":"§3.2"},{"comment":"The caption states that multi-label samples are reported once for each language, but the table does not show the number of multi-label instances; please clarify how the 5% and 6% multi-label rates are computed from the table, or add a multi-label row or column.","section":"Table 1"},{"comment":"The sigmoid threshold of 0.5 is selected based on the validation split; this is a free hyperparameter, and since SLIDE-fast's exact-match accuracy depends on it, please report its sensitivity or justify the choice with a small threshold sweep.","section":"§5.2"},{"comment":"The HeLI-OTS reference appears twice as Jauhiainen et al. (2022a) and (2022b) with identical titles and page ranges; these appear to be duplicate entries and should be consolidated.","section":"References"},{"comment":"The caption note 'NorBERT3 refers to the same model as SLIDE' is confusing because the table is about base-model selection; please clarify that the final model is named SLIDE-base.","section":"Table 3"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a worthwhile resource paper with a real new dataset and a clever silver-labeling trick, but the abstract's necessity claim is stronger than the evidence, and the annotation quality is under-documented. I'd send it to review, but with a request for IAA and a softened claim.\n\nWhat's actually new: a manually curated multi-label evaluation set for Danish, Bokmål, Nynorsk, Swedish, and the idea of using MT identity as a silver label (if the translator doesn't change the sentence, it's valid in the target language). The released models, especially the NorBERT3-based ones, beat strong baselines on that test set. The ablation study is honest and useful: punctuation augmentation helps a bit, NER swapping helps loose accuracy but hurts exact-match.\n\nSoft spots, in order of weight. First, the central claim—'multi-label identification is necessary for any accurate LID method'—is operationalized as exact-match on one test set of ~7k sentences. That's a single dataset, four languages, and the test set itself has no reported inter-annotator agreement. The annotators were all Norwegian, and the Limitations section admits their productive skills in Danish/Swedish are lower. The annotation conventions also inflate multi-label counts: proper-name-only sentences get all four labels, and Danish/Bokmål sentences differing only by an optional comma are labeled as both. Those choices are defensible, but they push the 5% multi-label rate up and directly penalize single-label baselines. Without agreement stats or a native-speaker check, the necessity claim is resting on a shaky foundation.\n\nSecond, the MT silver-labeling heuristic is never validated. The whole training pipeline assumes NorMistral stays conservative and only adds labels when the sentence is truly unchanged. A quick sanity check—e.g., measuring how often the MT model changes anything on known single-label data, or manually inspecting a sample of silver labels—would tell us a lot. The paper just asserts the observation.\n\nThird, the out-of-domain results are weaker: SLIDE-base underperforms GlotLID on the Haas & Derczynski test sets. The authors dismiss this with preprocessing caveats, but it undercuts the generalizability of the models.\n\nNone of this sinks the paper. The dataset alone is a contribution, and the models are useful for anyone doing Scandinavian NLP. The claims need to be scaled back and the annotation process needs more rigor. I'd recommend major revision, not rejection.","headline":"A useful dataset and models for multi-label Scandinavian LID, but the necessity claim exceeds the evidence and the annotation quality is under-documented.","tokens_in":13473,"tokens_out":2042,"would_cite":true,"duration_ms":16738,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SLIDE demonstrates that accurate language identification among Danish, Norwegian Bokmål, Nynorsk, and Swedish requires multi-label output, and its best model reaches 96.4% exact-match accuracy on a manually annotated test set.","keywords":["language identification","multi-label classification","Scandinavian languages","Norwegian Bokmål","Norwegian Nynorsk","closely related languages","silver labeling","machine translation"],"falsifier":"Randomly sample sentences from Scandinavian text, have native speakers of each of the four languages mark every language in which each sentence is acceptable, and compare the multi-valid share with the paper's 5% figure; if a large representative sample shows the share is near zero, the claim that accurate LID requires multi-label output loses its empirical footing.","tokens_in":12504,"feed_emoji":"🌐","tokens_out":10976,"duration_ms":94805,"temperature":0.7,"pith_summary":"SLIDE argues that sentence-level language identification (LID) among the four Mainland Scandinavian languages cannot be reduced to picking one label: a meaningful share of sentences—about 5% of the human-annotated test set and 16% of sentences shorter than six words—are valid in more than one of Danish, Bokmål, Nynorsk, and Swedish. Because of this, the paper claims, any accurate LID method must be able to output several languages at once, and single-label evaluation masks the failures. To train such a system without expensive manual labels, the authors convert machine translation into a labeling step: a sentence gets an extra language label whenever a translator leaves it unchanged. They release the resulting SLIDE dataset and models; the best model reaches 96.4% exact-match accuracy on the new test set, above all evaluated single-label baselines.","feed_headline":"Multi-label output is necessary for Scandinavian language ID","feed_subtitle":"A manually checked test set finds 5% of sentences belong to several languages; the new model beats all single-label baselines.","key_machinery":"The load-bearing mechanism is the translation-identity heuristic. The authors start from single-label sentences from existing treebank data, then translate each sentence into each of the other three target languages; whenever the translator returns the sentence unchanged, they add that language as a silver label. The heuristic relies on translation models' observed conservatism—they minimize edits—so an unchanged output is read as evidence that the source sentence is already acceptable in the target language. Training then proceeds as multi-label classification with binary cross-entropy on masked-language models pretrained for Norwegian (the SLIDE-xs/small/base models) and on an embedding-based model (SLIDE-fast). Two evaluation metrics carry the comparison: loose accuracy, which forgives a single-label model for choosing any gold label, and exact-match accuracy, which requires the full predicted label set to equal the gold set.","core_discovery":"The paper's central discovery is empirical and methodological at once. Empirically, manually re-examining treebank sentences shows that multi-label instances are not rare noise: they make up about 5% of the test set, and short sentences are disproportionately affected (16% under six words). Methodologically, the paper shows that a machine translation model can be used as a conservative annotator—if the translation of a sentence into another Scandinavian language changes nothing, the sentence is labeled as valid in that language—which lets a single-label corpus be extended into a multi-label training set at no manual cost. On the resulting SLIDE test set, the best multi-label model attains 96.4% exact-match accuracy, outperforming all evaluated baselines including the best single-label model, and the fast embedding-based variant matches the best baseline's exact-match accuracy while improving F1 on three of the four languages. The paper takes these results to show both that multi-label output is necessary for accurate LID among closely related languages and that its translation-based labeling method is sufficient to train such models.","pith_inferences":["The identity heuristic's reliability could be measured directly: sample silver-labeled sentences where the translator output was unchanged and ask native speakers whether the sentence is truly acceptable in the target language; that would give a noise estimate the paper does not provide.","The same multi-label framing should transfer to other tight language clusters such as Bosnian, Croatian, Montenegrin and Serbian, or Arabic dialects; the paper's necessity argument is not Scandinavia-specific, though the authors do not test these cases.","Because the training data is dominated by Bokmål and the annotation was done by Norwegian speakers, the consistently lower Danish F1 could partly reflect data skew and annotator asymmetry rather than an intrinsic property of Danish; a balanced Danish addition would be a cheap check.","The decision to ignore punctuation differences when labeling Danish and Bokmål ambiguity is a policy choice; systems for downstream tasks that care about orthographic correctness would need different labels."],"forward_implications":["LID benchmarks for closely related languages should use multi-label gold sets and exact-match accuracy; loose accuracy alone hides exactly the ambiguity the paper documents.","Off-the-shelf single-label LID tools will systematically mishandle the roughly 5% of multi-valid sentences, and much more so for short texts, where multi-validity reaches 16%.","The translation-identity labeling method can be reused for other closely related language groups whenever a reliable translator between those languages exists.","The SLIDE model family gives practitioners a clear speed-accuracy spectrum: the fast model suits high-throughput preprocessing, while the base model maximizes accuracy at about 96.4% exact-match accuracy."],"supporting_citations":[{"why":"Compiles the only earlier Nordic LID dataset and supplies the out-of-domain Wikipedia test splits used for generalization checks.","marker":"Haas and Derczynski (2021)"},{"why":"Provides the machine translation model whose output identity is the basis for silver-labeling multi-label training data.","marker":"Samuel et al. (2024)"},{"why":"Supplies the small high-quality parallel evaluation set used to fine-tune the translator in all language directions.","marker":"Tiedemann (2020)"},{"why":"Supplies the treebank corpora whose single-label sentences are extended into multi-label training and manually corrected into the SLIDE evaluation sets.","marker":"Nivre et al. (2016, 2020)"},{"why":"Provides the strongest baseline and the sentence embeddings used to build the fast SLIDE model.","marker":"Kargaran et al. (2023)"},{"why":"Provides the Norwegian masked-language models that are fine-tuned into the high-accuracy SLIDE variants.","marker":"Samuel et al. (2023)"},{"why":"Provides Heliport/HeLI-OTS, a fast published baseline that serves as the closest speed-accuracy competitor among existing systems.","marker":"Jauhiainen et al. (2022b)"}],"fun_headline_variants":["Multi-label output key for Scandinavian LID","Scandinavian LID: multi-label beats single-label","5% of Scandinavian sentences need multi-label","Translation-based labeling yields multi-label LID","Multi-label LID necessary, not optional"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that a machine translation model that leaves a sentence unchanged when translating it into another language reliably means the sentence is acceptable in that language; if translators are too conservative, the automatically generated multi-label training data is silently corrupted.","fun_headline_variants_meta":{"raw":{"variants":["Multi-label output key for Scandinavian LID","Scandinavian LID: multi-label beats single-label","5% of Scandinavian sentences need multi-label","Translation-based labeling yields multi-label LID","Multi-label LID necessary, not optional"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000932,"raw_usage":{"total_tokens":3944,"prompt_tokens":856,"completion_tokens":3088,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":3019}},"tokens_in":472,"tokens_out":3088,"duration_ms":19404,"temperature":1.0,"reasoning_tokens":3019,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T14:36:55.006280+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Randomly sample sentences from Scandinavian text, have native speakers of each of the four languages mark every language in which each sentence is acceptable, and compare the multi-valid share with the paper's 5% figure; if a large representative sample shows the share is near zero, the claim that accurate LID requires multi-label output loses its empirical footing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Compiles the only earlier Nordic LID dataset and supplies the out-of-domain Wikipedia test splits used for generalization checks."},{"cited_title":"Small Languages, Big Models: A Study of Continual Training on Languages of Norway","cited_arxiv_id":"2412.06484","evidence_quote":"Provides the machine translation model whose output identity is the basis for silver-labeling multi-label training data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the small high-quality parallel evaluation set used to fine-tune the translator in all language directions."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the strongest baseline and the sentence embeddings used to build the fast SLIDE model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Norwegian masked-language models that are fine-tuned into the high-accuracy SLIDE variants."}],"review_version":1}