Pith. sign in

REVIEW 2 major objections 6 minor 22 references

HiTZ at VarDial 2025 NorSID: Overcoming Data Scarcity with Language Transfer and Automatic Data Annotation

T0 review · 2 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read For Norwegian dialect assistant tasks, English-only training plus a small in-domain development set outperformed every combination that added machine-translated, tweeted, or transcribed data.

desk verdict A solid, honest shared-task paper whose main ranking is probably robust, but whose 'no dev-to-test drop' claim compares two different scorers and should be fixed. read the letter →

arxiv 2412.10095 v2 pith:TZCQQK5E submitted 2024-12-13 cs.CL cs.AIcs.LG

Jaione Bengoetxea , Mikel Zubillaga , Ekhi Azurmendi , Maite Heredia , Julen Etxaniz , Markel Ferro , Jeremy Barnes This is my paper · ORCID
classification cs.CLcs.AIcs.LG
keywords Norwegiandialectsintentdetectionslotfillingdialectidentificationcross-lingualtransfermultitasklearninglow-resourceNLPautomaticannotation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This system-description paper argues that, for low-resource spoken-language understanding in Norwegian dialects, the most reliable recipe is cross-lingual transfer from high-quality English data plus a small in-domain development set, rather than adding noisy machine-translated or automatically labeled dialect data. On the official test set, a multitask XLM-RoBERTa-large fine-tuned only on the English xSID training data achieved the best intent-and-slot Lambda average of 88.65%, and a NorBERT3-L fine-tuned only on the provided development set achieved the best dialect-identification weighted F1 of 84.17%. All combinations that added machine-translated Norwegian, tweets, or transcribed speech scored lower. The paper explains this through a narrow domain, intent labels that are predictable from word patterns, and a distribution mismatch between the available out-of-domain dialect corpora and the evaluation data.

What carries the argument

The argument rests on three components: a multitask fine-tuning setup that adds intent classification on the [CLS] token and slot tagging on each token, trained with the combined loss $L_{\text{total}} = L_{\text{slot}} \lambda + L_{\text{intent}}(1-\lambda)$ with $\lambda=0.7$; the xSID cross-lingual dataset, whose English training portion is human-annotated while other languages are machine-translated; and the Norwegian encoder NorBERT3-L for dialect identification, evaluated against a lexical-mapping SVM baseline and against models trained on automatically labeled tweets and transcriptions from NorDial, the Nordic Tweet Stream, the Nordic Dialect Corpus, and NB Samtale. To avoid data contamination from the 11-way paraphrases in NoMusic, the development set is split by original instance ID.

What would settle it

Take the English-only multitask model and evaluate it on a fresh, independently collected set of Norwegian virtual-assistant utterances covering the same intents, with slot annotations produced by separate native-speaker annotators; if the Lambda average drops materially below 88.65%, the result depends on the particular NoMusic translation and test split rather than on cross-lingual transfer.

Watch

Extended reading notes

Core claim

The paper's central discovery is that data quantity and language closeness matter less than data quality and domain match: English-only training outperforms all multilingual blends for intent and slot detection, and a Norwegian encoder fine-tuned on the development set alone beats every dialect-identification setup that uses tweets or transcriptions. The best intent-and-slot system is a multitask model with a weighted loss over XLM-RoBERTa-large, reaching a Lambda average of 88.65% on the test set (slot F1 85.37, intent accuracy 96.29), while the best dialect-identification system is NorBERT3-L fine-tuned on the dev-train-dev split, with weighted F1 84.17. The authors interpret these results as evidence that the manually translated NoMusic development set, despite its small size and skewed dialect distribution, is the most valuable resource for this task.

Load-bearing premise

The argument assumes that the NoMusic development and test annotations, produced by 11 human translators, are accurate and that the test set is an unbiased sample of the same narrow assistant domain; if the translations contain systematic errors or the test distribution differs, the reported scores and system rankings would not generalize.

Editorial extensions

If this is right

  • Dialectal spoken-language understanding can be built effectively by fine-tuning a multilingual encoder on high-quality English data alone; machine-translated Norwegian data lowered scores and is not a substitute.
  • Multitask training for intent detection and slot filling matches or slightly improves on single-task training while requiring only one model.
  • For dialect identification, a small in-domain development set is more valuable than large out-of-domain tweet or transcription corpora, with NorBERT3-L fine-tuned on dev-train-dev reaching the best test F1.
  • Intent detection in this narrow domain is nearly saturated, and models succeed via word-level patterns rather than deep semantic understanding, with Norwegian-pretrained models attaining top accuracy using fewer parameters.
  • When out-of-domain dialect data is used at all, phonetic transcriptions retain more dialectal features than orthographic ones, though neither beats in-domain fine-tuning.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that automatic annotation may pay off only when the unlabeled data matches the target domain; a testable extension is to evaluate the automatic-annotation pipeline on synthetic in-domain paraphrases rather than off-domain tweets.
  • The observation that intent classifiers rely on word patterns suggests they could degrade on paraphrase or code-mixed inputs, so an adversarial development set built by paraphrasing existing utterances would probe this weakness.
  • The data-scarcity lesson probably transfers to other dialectal SLU tasks: investing in a small high-quality parallel development set may matter more than collecting large noisy dialect corpora.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 6 minor

Summary. This paper presents the HiTZ team's submission to the NorSID shared task at VarDial 2025, covering intent detection, slot filling, and dialect identification in four Norwegian written dialects. For SID, the authors fine-tune a multitask XLM-RoBERTa-large model on different combinations of xSID languages; on the official test set the English-only model obtains the best Lambda average (88.65%), outperforming combinations that include machine-translated Norwegian, Danish, or Germanic languages. For dialect identification, they compare lexical-mapping SVMs, fine-tuned encoders, few-shot and fine-tuned decoders, and models trained on external tweet and transcription corpora; the best test result is NorBERT3-L fine-tuned on the shared-task development set (84.17 weighted F1). The paper also provides dataset analyses and discusses why cross-lingual transfer and automatically labeled external data did not help.

Significance. The reported results are practically useful: they suggest that, for a narrow-domain SLU task in low-resource dialects, a large English training set plus a small in-domain development set can beat elaborate combinations of machine-translated and automatically annotated data. The paper is transparent about its exploration of many alternative data sources and reports per-dialect F1, which helps the community understand where transfer fails. The main conclusions are tested against an external official test set and use established resources (xSID, NoMusic, NorDial, NTS, NB Samtale, NDC). The code is released, and the negative results on automatic annotation are a useful counterpoint to the common assumption that more data is always better. However, the validity of the "no drop" claim and the reliability of the dialect-ID winner are compromised by the scorer mismatch and post-hoc test-set comparisons described in the major comments.

major comments (2)
  1. [§3.3, footnote 4, Tables 3 and 5] The development-set SID scores (e.g., EN Lambda 84.96%) are computed with the authors' custom scorer, which does not apply the official post-processing, while the test-set scores (Lambda 88.65%) use the official VarDial scorer. The abstract's claim that "our models do not drop in performance compared to the development set" is therefore not supported by the reported numbers: the observed gains (slot F1 79.09 to 85.37; Lambda 84.96 to 88.65) could be an artifact of scoring differences. The authors should re-evaluate the development set with the official scorer or, at minimum, remove or substantially qualify the no-drop claim and the "similar distribution" explanation. This issue also affects the language-combination selection in Table 3, which was used to choose the submitted runs; although the ordering of the three submitted runs is the same under both scorers, the magnitude of the advantage and the no-drop conclusion are not verifiable as reported.
  2. [Table 12, §4.4] The dialect identification conclusion rests on comparing a large number of configurations on the official test set, and the best reported system (Dev-train-dev NorBERT3-L, 84.17 weighted F1) is selected from this table. All test values are single runs with no error bars, and many alternatives (NTS, NDC, NB Samtale, various few-shot and fine-tuned decoders) were scored on the same test set without any correction for multiple comparisons. The paper should state which systems were chosen before seeing the test labels, report variance across seeds, or explicitly frame the 84.17 result as the observed best in a post-hoc comparison rather than as a reliably superior system.
minor comments (6)
  1. [Footnote 4] The footnote does not explain what the official scorer's post-processing does; without this information, the reader cannot assess how large the dev-vs-test scorer difference might be.
  2. [§4.4.1, Table 12] The text says that few-shot models "perform worse than a random classifier (32.82)", but Table 12 reports the random classifier's test F1 as 32.40; this discrepancy should be corrected.
  3. [Figure 2, §3.3.1] The caption and text refer to models "trained for Intent Detection on the Norwegian train set" without specifying whether this is the machine-translated xSID-NB training data or a split of the NoMusic development set; please clarify.
  4. [§4.2.2, Table 12] The text says that preliminary experiments showed training on the full development set leads to the best performance, yet the best test result in Table 12 is Dev-train-dev, not Dev-train-dev-test; explain which model was selected before test labels were available and how the two variants relate.
  5. [§3.2, Eq. (1)] The multitask loss weight λ is set to 0.7 by intuition, and the Lambda average metric uses the same λ; no sensitivity analysis is reported, so the ranking of language combinations in Table 3 could depend on this arbitrary choice.
  6. [Table 5] The caption says "Best results in bold", but it is not clear in the typeset table whether the entire EN row or only the Lambda value is bold; please make the formatting explicit.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central SID and dialect-identification claims are evaluated on external test data and external resources, and the only self-citation (NorDial) is not load-bearing.

full rationale

The derivation chain is self-contained against external benchmarks. For Slot and Intent Detection, the models are trained on xSID (an external corpus) and evaluated on the official NorSID test set; the choice of the English-only model is made from development-set experiments (Table 3) and confirmed on the test set (Table 5), so the main result is an empirical comparison, not a fitted artifact. The Lambda average uses the same weight as the multitask loss, but this is an intentional evaluation design and does not make the test prediction equivalent to a training input. For Dialect Identification, the best system is NorBERT3-L fine-tuned on the development split and scored with the official evaluation script on the held-out test set (Section 4.4, Table 12); alternative data sources (NTS, NDC, NB Samtale) are evaluated on the same held-out dev-test/test sets and perform worse. The only self-citation is NorDial (Barnes et al., 2021), used as additional training/labeling data for those alternative experiments; because those experiments are explicitly outperformed by the development-only model, the paper's central conclusion does not depend on the correctness or validity of NorDial. The mismatch between the custom development scorer and the official test scorer (footnote 4, Section 3.3) is a methodological consistency concern about the 'no drop' claim, not a circularity: the dev and test numbers are not the same metric, but this does not reduce any prediction to its input by construction. No fitted parameter is renamed as a prediction, no uniqueness theorem is imported from the authors' prior work, and no ansatz is smuggled in via citation.

Assumptions & free parameters 8 free parameters · 6 assumptions · 0 invented entities

The central claims rest on the quality of the NoMusic annotations and the validity of the dialect labels, plus standard assumptions about pretrained transformer fine-tuning. No new entities are introduced. The main hand-chosen quantities are the multitask loss weight, hyperparameters, and heuristic data filtering choices.

free parameters (8)
  • Multitask loss weight lambda = 0.7
    Set by intuition that slot filling is harder than intent detection (Section 3.2, Eq. 1).
  • SID fine-tuning learning rate = 2e-5
    Chosen by hand; affects final model performance; listed in Appendix A.1.
  • SID batch size = 64
    Chosen by hand; listed in Appendix A.1.
  • SID number of epochs = 10
    Selected without explicit early stopping; listed in Appendix A.1.
  • NorBERT3-L dialect epochs = 15
    Selected based on development set performance; listed in Appendix A.2.
  • NDC minimum token lengths = 20 and 40
    Heuristic filter for short sentences with fewer dialectal traits (Section 4.3.1).
  • Downsampling target distribution for NTS and NDC = matched to dev distribution
    Automatically labeled instances are downsampled until class proportions match the development set; affects the trained model's prior (Section 4.3.2).
  • Lexical mapping dictionary = hand-crafted list
    Manually compiled mappings of pronouns, interrogatives, prepositions, verbal forms, and time expressions; basis for the Lexmap silver data (Section 4.2.1).
assumptions (6)
  • domain assumption The NoMusic dev and test manual translations and slot projections are accurate enough to serve as ground truth.
    All reported F1 scores rely on these labels; Section 2.1 describes construction by 11 translators.
  • domain assumption The four dialect labels (B, V, T, N) are mutually exclusive and sufficient for the test distribution.
    The tasks treat dialect identification as a 4-way classification (Section 2 and Table 2).
  • domain assumption Pretrained encoders and decoders provide representations that transfer to Norwegian dialect text.
    Fine-tuning success assumes XLM-RoBERTa-large, NorBERT3-L, NorMistral, and Llama 3.1 capture relevant dialectal features; Section 4.2.
  • domain assumption Machine-translated xSID Norwegian data preserves intent and slot labels well enough to be useful.
    The comparison of language combinations in Section 3.3 assumes translation noise is the reason for lower scores; if the translation were perfect, the ranking might differ.
  • domain assumption Semi-automatic mapping of NDC city locations to dialect labels and automatic classifier labeling are valid proxies for dialect identity.
    Used to create training data in Sections 4.3.1 and 4.3.2; if labels are wrong, conclusions about external data transfer are affected.
  • standard math Transformer fine-tuning via gradient descent minimizes the weighted cross-entropy loss in Eq. 1.
    Standard supervised learning assumption used throughout the experiments.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HiTZ at VarDial 2025 NorSID: Overcoming Data Scarcity with Language Transfer and Automatic Data Annotation." pith.science (2026). https://pith.science/paper/TZCQQK5E

@misc{pith2026241210095,
  author       = {Pith},
  title        = {Pith review of: HiTZ at VarDial 2025 NorSID: Overcoming Data Scarcity with Language Transfer and Automatic Data Annotation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TZCQQK5E}},
  note         = {Machine review of arXiv:2412.10095}
}
read the original abstract

In this paper we present our submission for the NorSID Shared Task as part of the 2025 VarDial Workshop (Scherrer et al., 2025), consisting of three tasks: Intent Detection, Slot Filling and Dialect Identification, evaluated using data in different dialects of the Norwegian language. For Intent Detection and Slot Filling, we have fine-tuned a multitask model in a cross-lingual setting, to leverage the xSID dataset available in 17 languages. In the case of Dialect Identification, our final submission consists of a model fine-tuned on the provided development set, which has obtained the highest scores within our experiments. Our final results on the test set show that our models do not drop in performance compared to the development set, likely due to the domain-specificity of the dataset and the similar distribution of both subsets. Finally, we also report an in-depth analysis of the provided datasets and their artifacts, as well as other sets of experiments that have been carried out but did not yield the best results. Additionally, we present an analysis on the reasons why some methods have been more successful than others; mainly the impact of the combination of languages and domain-specificity of the training data on the results.

Figures

Figures reproduced from arXiv: 2412.10095 by the authors.

Figure 1
Figure 1. The idea behind the multitask model fine [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Accuracy of pretrained English mod￾els (BERT, RoBERTa), multilingual models (XLM￾RoBERTa) and a Norwegian pretrained models (Nor￾BERT3) trained for Intent Detection on the Norwegian train set and evaluated the development set. 5Google’s 2020 BERT models were fine-tuned. 4 Dialect Identification In this section, we will present the dialect iden￾tification task, starting with the data used in our experiments (Section … view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

22 extracted references · 9 canonical work pages

  1. [1]

    e mi Aepli, C a g r C \

    No \"e mi Aepli, C a g r C \"o ltekin, Rob van der Goot, Tommi Jauhiainen, Mourhaf Kazzaz, Nikola Ljube s i \'c , Kai North, Barbara Plank, Yves Scherrer, and Marcos Zampieri. 2023. Findings of the V ar D ial evaluation campaign 2023. In Proceedings of the Tenth Workshop on NLP for Similar Languages, Varieties and Dialects, Dubrovnik, Croatia. Association...

    work page 2023

  2. [2]

    Jeremy Barnes, Petter M hlum, and Samia Touileb. 2021. Nordial: A preliminary corpus of written norwegian dialect use. NoDaLiDa 2021, page 445

    work page 2021

  3. [3]

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzm \' a n, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1911.02116 Unsupervised cross-lingual representation learning at scale . CoRR, abs/1911.02116

    arXiv 2019

  4. [4]

    Alice Coucke, Alaa Saade, Adrien Ball, Théodore Bluche, Alexandre Caulier, David Leroy, Clément Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Thibaut Lavril, Maël Primet, and Joseph Dureau. 2018. https://arxiv.org/abs/1805.10190 Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces . Prepri...

    arXiv 2018

  5. [5]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    doi:10.18653/v1/n19-1423 2019

  6. [6]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    arXiv 2024

  7. [7]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2023. https...

    doi:10.5281/zenodo.10256836 2023

  8. [8]

    Drew Harwell. 2018. https://www.washingtonpost.com/graphics/2018/business/alexa-does-not-understand-your-accent/ The accent gap . The Washington Post

    work page 2018

Show all 22 references
  1. [9]

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. https://arxiv.org/abs/2006.03654 Deberta: Decoding-enhanced bert with disentangled attention . Preprint, arXiv:2006.03654

    2021 arXiv

  2. [10]

    Janne Bondi Johannessen, Joel Priestley, Kristin Hagen, Tor Anders farli, and ystein Alexander Vangsnes. 2009. The nordic dialect corpus--an advanced research tool. In Proceedings of the 17th nordic conference of computational linguistics (nodalida 2009), pages 73--80

    2009

  3. [11]

    Janne Bondi Johannessen, Joel Priestley, Kristin Hagen, Anders N klestad, and Andr \'e Lynum. 2012. The nordic dialect corpus. In Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC'12), pages 3387--3391

    2012

  4. [12]

    Mikko Laitinen, Jonas Lundberg, Magnus Levin, and Rafael Messias Martins. 2018. The nordic tweet stream: A dynamic real-time monitor corpus of big and rich language data. In Digital Humanities in the Nordic Countries 3rd Conference, Helsinki, Finland, March 7-9, 2018, pages 34...

    2018

  5. [13]

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. https://arxiv.org/abs/1907.11692 Roberta: A robustly optimized bert pretraining approach . Preprint, arXiv:1907.11692

    2019 arXiv

  6. [14]

    Petter M hlum and Yves Scherrer. 2024. https://doi.org/10.18653/v1/2024.vardial-1.9 N o M usic - the N orwegian multi-dialectal slot and intent detection corpus . In Proceedings of the Eleventh Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2024), page...

    2024 doi

  7. [15]

    David Samuel, Andrey Kutuzov, Samia Touileb, Erik Velldal, Lilja vrelid, Egil R nningstad, Elina Sigdel, and Anna Palatkina. 2023. https://aclanthology.org/2023.nodalida-1.61 N or B ench -- a benchmark for N orwegian language models . In Proceedings of the 24th Nordic Conferen...

    2023

  8. [16]

    Yves Scherrer, Rob van der Goot, and Petter M hlum. 2025. V ar D ial evaluation campaign 2025: N orwegian slot and intent detection and dialect identification. In Proceedings of the Twelfth Workshop on NLP for Similar Languages, Varieties, and Dialects (VarDial 2025), Abu Dhab...

    2025

  9. [17]

    Sebastian Schuster, Sonal Gupta, Rushin Shah, and Mike Lewis. 2019. https://doi.org/10.18653/v1/N19-1380 Cross-lingual transfer learning for multilingual task oriented dialog . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computat...

    2019 doi

  10. [18]

    Iulia Turc, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Well-read students learn better: On the importance of pre-training compact models. arXiv preprint arXiv:1908.08962v2

    2019 arXiv

  11. [19]

    Rob van der Goot, Ibrahim Sharaf, Aizhan Imankulova, Ahmet \"U st \"u n, Marija Stepanovic, Alan Ramponi, Siti Oryza Khairunnisa, Mamoru Komachi, and Barbara Plank. 2021. From masked-language modeling to translation: Non- E nglish auxiliary tasks improve zero-shot spoken langu...

    2021

  12. [20]

    Miriam Winkler, Virginija Juozapaityte, Rob van der Goot, and Barbara Plank. 2024. Slot and intent detection resources for B avarian and L ithuanian: Assessing translations vs natural queries to digital assistants. In Proceedings of The 2024 Joint International Conference on C...

    2024

  13. [21]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  14. [22]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.