{"id":"00d26222-7286-4e55-8690-51efeafd423a","arxiv_id":"2501.03863","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Bavarian auxiliary tasks, especially NER, improve zero-shot slot and intent detection on Bavarian dialects, and intermediate-task training is more reliable than joint multi-task training.","lead":"The authors test whether adding Bavarian-language auxiliary tasks, such as named entity recognition and grammar tagging, improves zero-shot intent and slot detection for Bavarian dialects. Their best setup gains 5.1 points in intent accuracy and 8.4 points in slot F1, and they release a new Munich Bavarian evaluation dataset.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Auxiliary-task conclusions depend on a single PLM; prior work shows task effects vary across PLMs, so the claimed task ranking and gains may not generalize beyond mDeBERTa.","rationale":"The reader identified the single-PLM assumption as the weakest, and I agree. The paper is an empirical case study, but its abstract and conclusion make general claims ('NER having the most positive effect,' 'intermediate-task training yields more consistent performance gains'). These claims are only supported on mDeBERTa. Prior work cited in the paper (van der Goot et al., 2021a) explicitly shows auxiliary-task effects can differ across PLMs, so this is not a hypothetical risk. A second concern—that the best setup was selected after inspecting test results (Section 5: 'we select combinations that appear promising based on the results already obtained')—affects the magnitude of the reported gains, but the qualitative pattern is consistent across dialects and seeds, and the PLM concern is more fundamental: if the task ordering is model-specific, the central conclusions do not generalize regardless of model selection. A concrete replication with XLM-R would settle this. If the pattern holds, the conditional verdict can be upgraded; if not, the paper should be reframed as a single-model case study. Since the reader already conditioned on this, I recommend no change to the verdict.","tokens_in":22528,"tokens_out":6495,"duration_ms":57690,"concrete_test":"Run the key configurations—baseline SID, NER→SID, NER×SID, UD→SID, UD×SID, MLM→SID, MLM×SID, and MLM×NER→SID—with XLM-R-base (or mBERT) using the same MaChAmp hyperparameters, data splits, and three seeds, and compare delta-vs-baseline for intent accuracy and slot F1 on the three Bavarian test sets. If the ordering of auxiliary tasks and the ITT-over-MTL advantage replicate, the concern is resolved; if the ranking changes or gains disappear, the paper's conclusions must be re-scoped to mDeBERTa.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claims about auxiliary-task usefulness and training-schedule consistency are based entirely on experiments with mDeBERTa. Section 5 states: 'In the remaining setups, we only use mDeBERTa because of its strong performance as a baseline PLM (§6.1).' The Limitations section explicitly admits: 'we only carried out the (non-baseline) experiments with a single PLM and did not evaluate how robust the results are across PLMs.' This is not a minor caveat: van der Goot et al. (2021a), the work this paper builds on, found that syntactic auxiliary tasks improved slot filling for one PLM but harmed it for another (Section 2). If mDeBERTa's response to Bavarian auxiliary tasks is atypical, then the reported ordering (NER > UD > MLM, intermediate-task training more consistent than MTL) and the headline gains (+5.1 pp intents, +8.4 pp slots) are specific to this model and do not support the general conclusions in the abstract. The claim is load-bearing because the entire paper is an empirical generalization from one model; without evidence of cross-PLM robustness, the central argument is conditional.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies zero-shot slot and intent detection (SID) for Bavarian dialects by augmenting English SID fine-tuning with Bavarian auxiliary tasks. It compares joint multi-task learning (MTL) with intermediate-task training and three auxiliary task types: syntactic UD tasks (POS tagging and dependency parsing), named entity recognition (NER), and masked language modelling (MLM). The authors release a new Munich Bavarian test/development set (de-muc) and evaluate on three Bavarian test sets. Averaged over three seeds, they report that auxiliary tasks help slot filling more than intent classification, that NER is the most helpful auxiliary task, and that intermediate-task training is more consistent than MTL, with the best configuration (MLM×NER→SID) improving intent accuracy by 5.1 pp and slot F1 by 8.4 pp over the mDeBERTa baseline.","tokens_in":22724,"tokens_out":7075,"duration_ms":69264,"significance":"The paper makes a useful empirical and resource contribution to a genuinely low-resource area: Bavarian dialectal SID. The release of de-muc, the use of external Bavarian datasets, the multi-dialect evaluation, and the public code are concrete strengths. If the central findings are robust, the conclusion that target-variety auxiliary tasks, especially span-level NER, can improve zero-shot SID would be practically valuable and would complement the results of van der Goot et al. (2021a). However, the generalization of the findings is currently limited by three load-bearing issues: all non-baseline experiments use a single PLM, several reported differences are small relative to the reported seed variance and no significance testing is provided, and the best setup is selected using results that appear to include the test sets themselves.","major_comments":[{"comment":"All non-baseline auxiliary-task conclusions rest on a single PLM, mDeBERTa. Section 5 states: “In the remaining setups, we only use mDeBERTa because of its strong performance as a baseline PLM (§6.1).” The Limitations explicitly admit: “we only carried out the (non-baseline) experiments with a single PLM and did not evaluate how robust the results are across PLMs.” This is load-bearing because the paper itself cites van der Goot et al. (2021a), where the effect of syntactic auxiliary tasks differed across PLMs. The reported ranking NER > UD > MLM and the greater consistency of intermediate-task training over MTL may therefore be mDeBERTa-specific. I ask that the authors either run the key setups with at least one additional PLM (e.g., XLM-R or GBERT) or explicitly restrict the abstract and conclusion claims to mDeBERTa.","section":"Section 5 and Limitations"},{"comment":"The paper appears to select auxiliary-task combinations on the basis of results that include the test sets. Section 5 says: “we select combinations that appear promising based on the results already obtained,” and then the headline gains in Table 1 and the abstract are reported for the best of these combinations (MLM×NER→SID) on the same three Bavarian test sets. No held-out development set for model selection or multiple-comparison correction is described. This makes the +5.1 pp intent and +8.4 pp slot gains optimistic. The authors should either report selection on a truly held-out split or clearly frame the best-setup numbers as exploratory and selected from many configurations.","section":"Section 5 and Table 1"},{"comment":"The conclusion that MLM alone is harmful is confounded by the fact that the MLM auxiliary task was not learned properly in the MLM→SID setup. Table 8 reports a masked-token perplexity of 436.4 for MLM→SID, while the same MLM objective in MLM×NER→SID reaches 7.0. The footnote in Section 6.2 explicitly says “the auxiliary task was not learned properly,” and the Limitations attribute this to MaChAmp’s default epoch-level data splitting. Comparing a properly trained NER/UD auxiliary task against an under-trained MLM task does not support the claim that MLM is intrinsically harmful. The MLM experiment should be rerun with corrected training settings, or the MLM-based conclusions should be withdrawn and the abstract/§6.2 claims adjusted accordingly.","section":"Section 6.2, Table 8, and Limitations"},{"comment":"Several of the qualitative claims are based on differences that are small relative to the reported seed variance, yet no significance test or confidence interval is reported. For example, UD→SID improves intent accuracy by only +0.3 pp over the baseline, MLM×SID changes slot F1 by –0.7 pp, and even the NER intent gains (+2.7 to +3.0 pp) come from only three seeds per condition. With standard deviations of 2–4 pp on individual test sets, the ordering “NER > UD > MLM” and the claim that intermediate-task training “tends to beat” the baseline are not statistically established. I recommend paired bootstrap or a permutation test across seeds and test sets, or at least explicit confidence intervals, before drawing conclusions of this strength.","section":"Section 6.2 and Table 1"}],"minor_comments":[{"comment":"There is a typo: “tend be be worse” should read “tend to be worse.”","section":"Section 6.3"},{"comment":"The column headers are difficult to parse: the repeated “ITT MTL UD NER MLM” blocks make it unclear which differences correspond to which comparison. Please restructure the table or split it into separate panels for intents and slots.","section":"Table 1"},{"comment":"The metric name is inconsistent: Section 6.5 discusses “slot F1” while Table 9 labels the column “Slots (span F1, in %)”, whereas the rest of the paper uses “strict slot F1.” Please use one consistent term.","section":"Section 6.5 and Table 9"},{"comment":"In Table 3, the caption says “The similarities are calculates as 1 minus...” — “calculates” should be “calculated.”","section":"Appendix A"},{"comment":"Using lines to connect points on a categorical x-axis may imply interpolation between language varieties; points or boxplots with the same grouping would present the comparison more accurately.","section":"Figure 4"},{"comment":"The de-muc dataset is translated and annotated by a single native speaker, which is a real limitation for a benchmark. I appreciate that this is openly stated in the data statement (B.11), but it should also be mentioned in the main text when the dataset is introduced, not only in the appendix.","section":"Section 4.1 and Appendix B.11"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical contribution to a genuinely low-resource area, and the de-muc dataset is valuable. The main risk is that the central conclusions are presented more generally than the evidence supports. I would encourage the editor to require either a second PLM for the key auxiliary-task setups or a clear reframing of the claims as mDeBERTa-specific, together with a treatment of the model-selection and significance issues. If the authors add a second PLM and hold out a proper selection split, the paper could become a strong acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. The paper is a careful, honest empirical study of using Bavarian auxiliary tasks to improve zero-shot slot and intent detection on Bavarian dialects, and it releases a new Munich Bavarian SID test set. The headline finding — NER is the most helpful auxiliary task and intermediate-task training is more consistent than joint multi-task learning — is plausible but rests on a single PLM, mDeBERTa, which the authors admit in the Limitations. That makes the central generalization conditional, not wrong.\n\nWhat is actually new: the de-muc dataset, a systematic comparison of UD, NER, and MLM as auxiliary tasks in both MTL and intermediate-task setups, plus evaluation on additional naturalistic and MASSIVE-derived Bavarian sets. The paper does several things well: three random seeds with standard deviations, per-dialect breakdowns, a data statement, and a Limitations section that names the single-PLM issue, the MLM hyperparameter problem, and the softmax decoder. The pattern holds across three Bavarian test sets and mostly transfers to Swiss German, which is decent evidence the effect is not an accident of one dataset.\n\nThe main soft spot is the one the authors flag: all non-baseline conclusions use mDeBERTa. van der Goot et al. (2021a), which this line of work builds on, found that auxiliary task effects differ across PLMs. So the task ranking (NER > UD > MLM) and the +5.1/+8.4 pp gains are established for mDeBERTa only. A second PLM, even just XLM-R, would have made the claims much stronger. Two minor issues: the best configurations were selected after seeing test results, so the headline numbers are partly cherry-picked, and the new dataset is not yet downloadable. The MLM setup is also an oddity — mDeBERTa was pre-trained with RTD, not MLM, and the authors acknowledge the MLM task was not learned well with default hyperparameters.\n\nThis is a useful paper for researchers working on dialectal or low-resource SID, and for anyone comparing MTL to intermediate-task training. It is not a conceptual breakthrough but it is a competent, reproducible empirical contribution with an honest limitations section. The single-PLM issue is real, but the paper does not hide it. I would send it to peer review and ask the authors to add cross-PLM evidence and release the dataset. A strong referee could turn this into a solid paper.","headline":"Careful empirical study of Bavarian auxiliary tasks for dialectal SID; the new dataset and honest limitations are real assets, but the central task ranking rests on one PLM.","tokens_in":23270,"tokens_out":2217,"would_cite":true,"duration_ms":19728,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Training on Bavarian named-entity recognition before English slot-and-intent data improves intent accuracy by 5.1 points and slot F1 by 8.4 points over a strong baseline, with intermediate-task training more reliable than joint multi-task…","keywords":["slot and intent detection","Bavarian dialects","zero-shot cross-lingual transfer","intermediate-task training","multi-task learning","named entity recognition","auxiliary tasks","dialectal NLP"],"falsifier":"Run the same auxiliary-task setups with a second multilingual encoder (e.g., XLM-R or mBERT) on the same three Bavarian test sets; if NER no longer gives the largest gains or intermediate-task training no longer beats joint multi-task learning, then the paper's central ranking is specific to mDeBERTa rather than a general property of dialectal auxiliary tasks.","tokens_in":22295,"feed_emoji":"🗣️","tokens_out":4355,"duration_ms":38441,"temperature":0.7,"pith_summary":"The paper asks whether cheap, lightly annotated Bavarian data can make a slot-and-intent model work for Bavarian dialects when no Bavarian training data for the target task exists. It answers yes: using Bavarian auxiliary tasks during fine-tuning improves zero-shot transfer from English SID training data to three Bavarian test sets. Named-entity recognition is the most consistently helpful auxiliary task, and sequential intermediate-task training beats simultaneous multi-task learning. The best setup, MLM×NER→SID, raises intent classification accuracy by 5.1 and strict slot F1 by 8.4 percentage points. The gains also carry over to naturalistic and translated Bavarian data from the same region.","feed_headline":"Bavarian auxiliary tasks lift zero-shot intent detection by 5.1 points","feed_subtitle":"Named-entity recognition first, then English slot-intent data, beats the baseline on three Bavarian dialects.","key_machinery":"The central object is the fine-tuning scheme rather than a single mathematical identity: a pre-trained multilingual encoder (mDeBERTa v.3) fine-tuned on English SID data, optionally preceded by or combined with Bavarian auxiliary tasks. Task heads are simple softmax heads for sequence labelling and classification, trained with equally weighted losses inside a multi-task fine-tuning toolkit. The comparison that carries the argument is between joint multi-task training (auxiliary task × SID) and intermediate-task training (auxiliary task → SID), holding the backbone and hyperparameters fixed. The auxiliary data are the MaiBaam UD treebank (POS and dependency parsing), BarNER (named-entity spans), and a Bavarian Wikipedia MLM subset. The mechanism the authors point to is task similarity: NER and slots are both token-level span labelling, so NER transfers best; MLM and syntactic tasks are less aligned with slot filling and only help when combined or ordered appropriately.","core_discovery":"The central claim is that auxiliary-task fine-tuning on Bavarian data improves zero-shot slot and intent detection for Bavarian dialects, and that the choice of auxiliary task and training order matters more than the sheer amount of auxiliary data. NER is the most beneficial auxiliary task because it is token-level and structurally similar to slot filling; syntactic UD tasks help only when used as an intermediate task, and MLM alone hurts but helps when combined with NER. Intermediate-task training, in which the model is first fine-tuned on Bavarian auxiliary tasks and only afterwards on English SID data, produces consistent gains, whereas joint multi-task training can severely degrade intent classification when syntactic tasks are included. The best model, trained first jointly on MLM and NER and then on SID, beats the mDeBERTa baseline by 5.1 percentage points in intent accuracy and 8.4 percentage points in strict slot F1, averaged over the three Bavarian test sets. These are the paper's results, reported as means over three random seeds.","pith_inferences":["If task similarity is the driver, then other token-level Bavarian tasks, such as morphological tagging or chunking, may give gains comparable to NER for slot filling; this is a testable extension the paper does not run.","Because all auxiliary-task experiments use a single pre-trained language model, the ranking of tasks and training orders could shift on other backbones; a replication on XLM-R or mBERT would show whether the NER-first recipe is general.","The MLM-alone results may reflect data size rather than the task itself, since only 1,500 Bavarian sentences are used; scaling up dialectal MLM and retesting could change the conclusion that MLM alone hurts.","For a practitioner, the implied recipe is: collect or reuse dialectal NER and raw text, fine-tune on those first, then on a high-resource SID corpus, and expect the largest gains on slot-heavy queries rather than intent classification."],"forward_implications":["A Bavarian NER dataset can serve as a low-cost substitute for Bavarian SID training data in zero-shot settings, improving both slot and intent performance.","Ordering matters: training on auxiliary tasks before SID is more reliable than training them jointly with SID, and joint training with UD can collapse intent accuracy by tens of points.","Auxiliary tasks improve slot filling more than intent classification, so applications that depend on precise slot values benefit most from this recipe.","The gains generalize within the same dialect region: the best model improves intent accuracy by 6.7–7.9 percentage points and slot F1 by 9.7–9.9 points on naturalistic and MASSIVE-translated Upper Bavarian test sets.","The newly released Munich Bavarian (de-muc) test set offers a stricter intra-dialectal check, since scores there are often lower than on the other Central Bavarian sets even though trends match."],"supporting_citations":[{"why":"Supplies the xSID English SID training data, the de-st Bavarian test set, and the MTL-with-auxiliary-tasks setup that this paper extends.","marker":"van der Goot et al. (2021a)"},{"why":"Supplies the de-ba xSID extension and the additional naturalistic and MASSIVE-translated Upper Bavarian evaluation sets used for robustness checks.","marker":"Winkler et al. (2024)"},{"why":"Supplies BarNER, the Bavarian named-entity recognition dataset that turns out to be the most helpful auxiliary task.","marker":"Peng et al. (2024)"},{"why":"Supplies MaiBaam, the Bavarian Universal Dependencies treebank used for the syntactic auxiliary tasks (POS tagging and dependency parsing).","marker":"Blaschke et al. (2024a)"},{"why":"Supplies the pre-processed Bavarian Wikipedia subset used for the masked language modelling auxiliary task.","marker":"Artemova and Plank (2023)"},{"why":"Provides mDeBERTa v.3, the encoder used for all auxiliary-task experiments.","marker":"He et al. (2021a)"},{"why":"Provides the intermediate-task training procedure that the paper adapts for the auxiliary-then-SID setups.","marker":"Pruksachatkun et al. (2020)"},{"why":"Supplies the Bernese Swiss German test data used to check whether Bavarian auxiliary-task trends extend to a related Upper German dialect.","marker":"Aepli et al. (2023)"}],"fun_headline_variants":["NER auxiliary task lifts Bavarian intent and slot performance","Intermediate training beats multi-task for Bavarian SID","Bavarian dialects: NER-first training gains 5.1 intent, 8.4 slot F1","Auxiliary tasks improve zero-shot detection in Bavarian dialects","Best Bavarian SID: MLM+NER then English data, gains 8.4 F1"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All non-baseline conclusions rest on a single pre-trained language model, mDeBERTa, chosen for its strong baseline performance; if auxiliary tasks affect that model differently than other multilingual encoders, the ranking of tasks and training orders may not generalize.","fun_headline_variants_meta":{"raw":{"variants":["NER auxiliary task lifts Bavarian intent and slot performance","Intermediate training beats multi-task for Bavarian SID","Bavarian dialects: NER-first training gains 5.1 intent, 8.4 slot F1","Auxiliary tasks improve zero-shot detection in Bavarian dialects","Best Bavarian SID: MLM+NER then English data, gains 8.4 F1"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000251,"raw_usage":{"total_tokens":1559,"prompt_tokens":947,"completion_tokens":612,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":563,"completion_tokens_details":{"reasoning_tokens":510}},"tokens_in":563,"tokens_out":612,"duration_ms":5554,"temperature":1.0,"reasoning_tokens":510,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:45:47.303077+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same auxiliary-task setups with a second multilingual encoder (e.g., XLM-R or mBERT) on the same three Bavarian test sets; if NER no longer gives the largest gains or intermediate-task training no longer beats joint multi-task learning, then the paper's central ranking is specific to mDeBERTa rather than a general property of dialectal auxiliary tasks.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the de-ba xSID extension and the additional naturalistic and MASSIVE-translated Upper Bavarian evaluation sets used for robustness checks."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the pre-processed Bavarian Wikipedia subset used for the masked language modelling auxiliary task."}],"review_version":1}