{"id":"87eb8e30-e845-43a2-bb28-0dc3d82f4439","arxiv_id":"2412.10095","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"For the NorSID shared task, English-only cross-lingual training and a Norwegian model fine-tuned on the small development set outperformed language transfer and automatic data annotation.","lead":"This paper reports a team's results in a contest to identify intents, slots, and dialects in Norwegian assistant queries. The winning systems used English-only training for intent and slot detection, and a Norwegian model fine-tuned on the small development set for dialect identification.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SID dev-set scores use a custom scorer, not the official one; the 'no test drop' claim compares incompatible metrics.","rationale":"The reader's weakest assumption concerned annotation quality and test-set bias in NoMusic. My review identifies a more immediate and concrete threat to the central empirical claim: the development-set scores in Table 3 were produced with a custom scorer, while the test-set scores in Table 5 use the official shared-task scorer. Because these scorers differ (as acknowledged in footnote 4), any dev-versus-test comparison in the paper is invalid. This directly undermines the abstract's 'no drop' claim and the related inference about distributional similarity, which is one of the paper's stated contributions. However, the primary ranking of submitted systems on the official test set is likely unaffected because the same ordering appears in both the custom dev scorer and the official test scorer, and the margins on the test set are large for the dialect task and substantial for the SID Lambda average. The paper has real strengths: reproducible code, transparent reporting of many negative results, and a useful analysis of why language transfer and automatic annotation failed. The appropriate remedy is a condition: recompute dev scores with the official scorer and re-evaluate the 'no drop' claim, and clarify which dialect model was actually submitted. This does not change the reader's conditional verdict, but the condition should explicitly include the scorer inconsistency.","tokens_in":12327,"tokens_out":6909,"duration_ms":70231,"concrete_test":"Re-run the development set through the official VarDial evaluation script for the three submitted SID models (EN, GER-NB, EN+NB) and for the language-combination grid in Table 3. If the official dev Lambda averages are close to the Table 5 test values, the 'no drop' claim holds; if official dev scores already reach about 88 Lambda for EN, the test actually dropped and the paper's interpretation fails. Also check whether the ranking under the official dev scorer matches the Table 5 ordering; if not, the model selection was suboptimal.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SID claim rests on a comparison between development and test scores that is not apples-to-apples. Table 3 (dev) is computed with the authors' own scorer (footnote 4, Section 3.3), while Table 5 (test) uses the official VarDial scorer. The paper's statement that 'our models do not drop in performance compared to the development set' (Abstract, Section 5) and the inference that dev/test distributions are similar are therefore unsupported: the large test-vs-dev gains (slot F1 79.09 to 85.37; Lambda 84.96 to 88.65) could be entirely an artifact of post-processing differences between scorers. Moreover, model selection among language combinations (Section 3.2, Table 3) was done with the custom scorer; if the official scorer ranks candidates differently, the submitted runs may not be the actual best under the official metric. The ordering of the three submitted runs is the same in both scorers, so the headline result (EN-only > GER-NB > EN+NB) is probably robust, but the magnitude of the advantage and the 'no drop' conclusion are not verifiable from the reported numbers.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents the HiTZ team's submission to the NorSID shared task at VarDial 2025, covering intent detection, slot filling, and dialect identification in four Norwegian written dialects. For SID, the authors fine-tune a multitask XLM-RoBERTa-large model on different combinations of xSID languages; on the official test set the English-only model obtains the best Lambda average (88.65%), outperforming combinations that include machine-translated Norwegian, Danish, or Germanic languages. For dialect identification, they compare lexical-mapping SVMs, fine-tuned encoders, few-shot and fine-tuned decoders, and models trained on external tweet and transcription corpora; the best test result is NorBERT3-L fine-tuned on the shared-task development set (84.17 weighted F1). The paper also provides dataset analyses and discusses why cross-lingual transfer and automatically labeled external data did not help.","tokens_in":12731,"tokens_out":8052,"duration_ms":84474,"significance":"The reported results are practically useful: they suggest that, for a narrow-domain SLU task in low-resource dialects, a large English training set plus a small in-domain development set can beat elaborate combinations of machine-translated and automatically annotated data. The paper is transparent about its exploration of many alternative data sources and reports per-dialect F1, which helps the community understand where transfer fails. The main conclusions are tested against an external official test set and use established resources (xSID, NoMusic, NorDial, NTS, NB Samtale, NDC). The code is released, and the negative results on automatic annotation are a useful counterpoint to the common assumption that more data is always better. However, the validity of the \"no drop\" claim and the reliability of the dialect-ID winner are compromised by the scorer mismatch and post-hoc test-set comparisons described in the major comments.","major_comments":[{"comment":"The development-set SID scores (e.g., EN Lambda 84.96%) are computed with the authors' custom scorer, which does not apply the official post-processing, while the test-set scores (Lambda 88.65%) use the official VarDial scorer. The abstract's claim that \"our models do not drop in performance compared to the development set\" is therefore not supported by the reported numbers: the observed gains (slot F1 79.09 to 85.37; Lambda 84.96 to 88.65) could be an artifact of scoring differences. The authors should re-evaluate the development set with the official scorer or, at minimum, remove or substantially qualify the no-drop claim and the \"similar distribution\" explanation. This issue also affects the language-combination selection in Table 3, which was used to choose the submitted runs; although the ordering of the three submitted runs is the same under both scorers, the magnitude of the advantage and the no-drop conclusion are not verifiable as reported.","section":"§3.3, footnote 4, Tables 3 and 5"},{"comment":"The dialect identification conclusion rests on comparing a large number of configurations on the official test set, and the best reported system (Dev-train-dev NorBERT3-L, 84.17 weighted F1) is selected from this table. All test values are single runs with no error bars, and many alternatives (NTS, NDC, NB Samtale, various few-shot and fine-tuned decoders) were scored on the same test set without any correction for multiple comparisons. The paper should state which systems were chosen before seeing the test labels, report variance across seeds, or explicitly frame the 84.17 result as the observed best in a post-hoc comparison rather than as a reliably superior system.","section":"Table 12, §4.4"}],"minor_comments":[{"comment":"The footnote does not explain what the official scorer's post-processing does; without this information, the reader cannot assess how large the dev-vs-test scorer difference might be.","section":"Footnote 4"},{"comment":"The text says that few-shot models \"perform worse than a random classifier (32.82)\", but Table 12 reports the random classifier's test F1 as 32.40; this discrepancy should be corrected.","section":"§4.4.1, Table 12"},{"comment":"The caption and text refer to models \"trained for Intent Detection on the Norwegian train set\" without specifying whether this is the machine-translated xSID-NB training data or a split of the NoMusic development set; please clarify.","section":"Figure 2, §3.3.1"},{"comment":"The text says that preliminary experiments showed training on the full development set leads to the best performance, yet the best test result in Table 12 is Dev-train-dev, not Dev-train-dev-test; explain which model was selected before test labels were available and how the two variants relate.","section":"§4.2.2, Table 12"},{"comment":"The multitask loss weight λ is set to 0.7 by intuition, and the Lambda average metric uses the same λ; no sensitivity analysis is reported, so the ranking of language combinations in Table 3 could depend on this arbitrary choice.","section":"§3.2, Eq. (1)"},{"comment":"The caption says \"Best results in bold\", but it is not clear in the typeset table whether the entire EN row or only the Lambda value is bold; please make the formatting explicit.","section":"Table 5"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a shared-task system description rather than a methodological advance, and the journal should judge whether such papers fit its scope. Methodologically, the central empirical findings are interesting and the extensive negative results are a strength, but the scorer mismatch and the post-hoc selection over the official test set need to be addressed before publication. There is no meaningful circularity: the main conclusions depend on the external official test set and on external resources, and the only self-citation (NorDial) is used as training data, which is legitimate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a competent system paper for the NorSID shared task, and the central ranking result is probably right. The paper's real value is in the negative results: for intent and slot detection, adding machine-translated Norwegian or other Germanic languages to English xSID hurts performance, and for dialect identification, fine-tuning NorBERT3-L on the development data alone beats every automatic annotation setup they tried. Those are useful, clean findings for anyone working on low-resource Norwegian dialect SLU, and the code is public.\n\nThe paper is also honest about experiments that didn't work, which is rarer than it should be.\n\nThe main soft spot is real and appears in the abstract. Dev-set scores in Section 3.3 (Table 3) are computed with the authors' own scorer, no post-processing, per footnote 4. Test scores in Table 5 use the official VarDial scorer. The abstract's claim that models 'do not drop in performance compared to the development set' is therefore not supported—the two numbers are not the same metric. The slot F1 jump from 79.09 to 85.37 and the Lambda jump from 84.96 to 88.65 could be partly a scorer artifact. That said, the ranking of the three submitted runs is the same under both scorers, so the headline conclusion (EN-only > GER-NB > EN+NB) likely survives. But the magnitude and the 'no drop' claim need to be either verified by rescoring dev with the official script or softened.\n\nSecond soft spot: Table 12 reports dev and test F1 for many configurations, meaning the test set was used repeatedly after the fact. That's common in shared task papers, but it should be labeled as post-hoc evaluation, since the multiple comparisons inflate the chance of a spurious winner. Minor points: no error bars on test results, and the 'word patterns rather than semantics' interpretation in Section 3.3.1 is plausible but is supported only by the accuracy curve in Figure 2.\n\nOverall, the math and data handling are fine aside from the scorer mismatch. The paper deserves a serious referee. I'd ask the authors to fix the dev/test comparison, either with the official scorer or with a caveat, and to add a line about post-hoc test evaluation. Then it's a solid workshop paper.","headline":"A solid, honest shared-task paper whose main ranking is probably robust, but whose 'no dev-to-test drop' claim compares two different scorers and should be fixed.","tokens_in":13129,"tokens_out":2583,"would_cite":true,"duration_ms":25617,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"For Norwegian dialect assistant tasks, English-only training plus a small in-domain development set outperformed every combination that added machine-translated, tweeted, or transcribed data.","keywords":["Norwegian dialects","intent detection","slot filling","dialect identification","cross-lingual transfer","multitask learning","low-resource NLP","automatic annotation"],"falsifier":"Take the English-only multitask model and evaluate it on a fresh, independently collected set of Norwegian virtual-assistant utterances covering the same intents, with slot annotations produced by separate native-speaker annotators; if the Lambda average drops materially below 88.65%, the result depends on the particular NoMusic translation and test split rather than on cross-lingual transfer.","tokens_in":12170,"feed_emoji":"🗣️","tokens_out":6455,"duration_ms":64973,"temperature":0.7,"pith_summary":"This system-description paper argues that, for low-resource spoken-language understanding in Norwegian dialects, the most reliable recipe is cross-lingual transfer from high-quality English data plus a small in-domain development set, rather than adding noisy machine-translated or automatically labeled dialect data. On the official test set, a multitask XLM-RoBERTa-large fine-tuned only on the English xSID training data achieved the best intent-and-slot Lambda average of 88.65%, and a NorBERT3-L fine-tuned only on the provided development set achieved the best dialect-identification weighted F1 of 84.17%. All combinations that added machine-translated Norwegian, tweets, or transcribed speech scored lower. The paper explains this through a narrow domain, intent labels that are predictable from word patterns, and a distribution mismatch between the available out-of-domain dialect corpora and the evaluation data.","feed_headline":"English-only data wins Norwegian assistant tasks","feed_subtitle":"Best scores came from a multitask English model and a Norwegian model fine-tuned on the dev set alone.","key_machinery":"The argument rests on three components: a multitask fine-tuning setup that adds intent classification on the [CLS] token and slot tagging on each token, trained with the combined loss $L_{\\text{total}} = L_{\\text{slot}} \\lambda + L_{\\text{intent}}(1-\\lambda)$ with $\\lambda=0.7$; the xSID cross-lingual dataset, whose English training portion is human-annotated while other languages are machine-translated; and the Norwegian encoder NorBERT3-L for dialect identification, evaluated against a lexical-mapping SVM baseline and against models trained on automatically labeled tweets and transcriptions from NorDial, the Nordic Tweet Stream, the Nordic Dialect Corpus, and NB Samtale. To avoid data contamination from the 11-way paraphrases in NoMusic, the development set is split by original instance ID.","core_discovery":"The paper's central discovery is that data quantity and language closeness matter less than data quality and domain match: English-only training outperforms all multilingual blends for intent and slot detection, and a Norwegian encoder fine-tuned on the development set alone beats every dialect-identification setup that uses tweets or transcriptions. The best intent-and-slot system is a multitask model with a weighted loss over XLM-RoBERTa-large, reaching a Lambda average of 88.65% on the test set (slot F1 85.37, intent accuracy 96.29), while the best dialect-identification system is NorBERT3-L fine-tuned on the dev-train-dev split, with weighted F1 84.17. The authors interpret these results as evidence that the manually translated NoMusic development set, despite its small size and skewed dialect distribution, is the most valuable resource for this task.","pith_inferences":["The paper leaves implicit that automatic annotation may pay off only when the unlabeled data matches the target domain; a testable extension is to evaluate the automatic-annotation pipeline on synthetic in-domain paraphrases rather than off-domain tweets.","The observation that intent classifiers rely on word patterns suggests they could degrade on paraphrase or code-mixed inputs, so an adversarial development set built by paraphrasing existing utterances would probe this weakness.","The data-scarcity lesson probably transfers to other dialectal SLU tasks: investing in a small high-quality parallel development set may matter more than collecting large noisy dialect corpora."],"forward_implications":["Dialectal spoken-language understanding can be built effectively by fine-tuning a multilingual encoder on high-quality English data alone; machine-translated Norwegian data lowered scores and is not a substitute.","Multitask training for intent detection and slot filling matches or slightly improves on single-task training while requiring only one model.","For dialect identification, a small in-domain development set is more valuable than large out-of-domain tweet or transcription corpora, with NorBERT3-L fine-tuned on dev-train-dev reaching the best test F1.","Intent detection in this narrow domain is nearly saturated, and models succeed via word-level patterns rather than deep semantic understanding, with Norwegian-pretrained models attaining top accuracy using fewer parameters.","When out-of-domain dialect data is used at all, phonetic transcriptions retain more dialectal features than orthographic ones, though neither beats in-domain fine-tuning."],"supporting_citations":[{"why":"Supplies the xSID dataset, whose English training data is the training source for the best intent-and-slot model.","marker":"(van der Goot et al., 2021)"},{"why":"Introduces NoMusic, the development and test data for all three tasks and defines the four Norwegian dialects and annotation setup.","marker":"(Mæhlum and Scherrer, 2024)"},{"why":"Provides XLM-RoBERTa-large, the encoder backbone for the multitask intent-and-slot model and a dialect-identification baseline.","marker":"(Conneau et al., 2019)"},{"why":"Provides NorBERT3-L, the Norwegian pretrained encoder that yields the best dialect-identification results.","marker":"(Samuel et al., 2023)"},{"why":"Supplies NorDial tweets used as an alternative dialect-training source and as training data for the NorDial classifier in the automatic annotation pipeline.","marker":"(Barnes et al., 2021)"},{"why":"Supplies the Nordic Tweet Stream tweets used as automatically annotated training data for dialect identification.","marker":"(Laitinen et al., 2018)"},{"why":"Supplies the Nordic Dialect Corpus transcriptions used for semi-automatically and automatically labeled dialect training.","marker":"(Johannessen et al., 2009, 2012)"}],"fun_headline_variants":["English-only training beats multilingual for Norwegian tasks","Data quality trumps quantity in Norwegian dialect ID","Tiny dev set outperforms tweet data for Norwegian","Domain-specific data wins over size in NorSID"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that the NoMusic development and test annotations, produced by 11 human translators, are accurate and that the test set is an unbiased sample of the same narrow assistant domain; if the translations contain systematic errors or the test distribution differs, the reported scores and system rankings would not generalize.","fun_headline_variants_meta":{"raw":{"variants":["English-only training beats multilingual for Norwegian tasks","Data quality trumps quantity in Norwegian dialect ID","Tiny dev set outperforms tweet data for Norwegian","Domain-specific data wins over size in NorSID"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000316,"raw_usage":{"total_tokens":1792,"prompt_tokens":949,"completion_tokens":843,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":565,"completion_tokens_details":{"reasoning_tokens":783}},"tokens_in":565,"tokens_out":843,"duration_ms":9458,"temperature":1.0,"reasoning_tokens":783,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:21:11.458726+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the English-only multitask model and evaluate it on a fresh, independently collected set of Norwegian virtual-assistant utterances covering the same intents, with slot annotations produced by separate native-speaker annotators; if the Lambda average drops materially below 88.65%, the result depends on the particular NoMusic translation and test split rather than on cross-lingual transfer.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the xSID dataset, whose English training data is the training source for the best intent-and-slot model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies NorDial tweets used as an alternative dialect-training source and as training data for the NorDial classifier in the automatic annotation pipeline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Nordic Tweet Stream tweets used as automatically annotated training data for dialect identification."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Nordic Dialect Corpus transcriptions used for semi-automatically and automatically labeled dialect training."}],"review_version":1}