{"id":"efc81e00-6adb-40ab-9acf-9d04650983be","arxiv_id":"2608.05510","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Perturbation-based continued pre-training reliably improves zero-shot dialect robustness, with character noise strongest overall, and different perturbation types improve performance via distinct mechanisms.","lead":"This paper tests six ways of continuing to train a large language model on slightly altered text to help it understand dialects it has never seen. It finds that all methods help, character-level changes help most, and different methods build robustness through different internal changes.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Figure 2's 'distinct mechanisms' claim rests on correlations whose sample size is never reported; the stated p-values are arithmetically incompatible with the three aggregate points in Tables 13–14.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the mechanism attribution depends on correlations whose observation set is underspecified and whose reported p-values are inconsistent with the only aggregate tables shown. My read of the full text confirms this. The primary robustness finding—that perturbation-based CPT, especially CHAR, improves zero-shot dialect performance—is well supported by Table 3, although the paper's claim of '43 of 45' outperforming BASE is actually '42 of 45' on the reported values: CLEAN ties BASE on German NER, PHON ties BASE on Italian NER, and PHON underperforms BASE on Arabic NER. This count error is secondary and does not overturn the general trend. The distinct-mechanisms claim, however, is the novel contribution and is not verifiable from the manuscript as written. The reported p-values cannot come from the three aggregate rows in Tables 13 and 14; they would be plausible for per-dialect observations, but the paper neither states this nor presents the per-dialect values. A conditional verdict requiring the authors to specify the observation set and provide the underlying per-dialect data is appropriate. No stronger rejection is warranted because the per-dialect analysis may well support the p-values, and the main robustness result stands.","tokens_in":14918,"tokens_out":5136,"duration_ms":46087,"concrete_test":"Reconstruct the exact observation set for Figure 2 from per-dialect (not macro-averaged) BPC and cosine-similarity values for the three parallel settings: German intent (3 dialect varieties), Italian intent (1 dialect variety), and Arabic topic (7 dialect varieties), totaling n=11, or request the authors' data. Recomputed each Pearson r and p-value. If the reported r/p pairs do not match the n=11 computation, or if the authors cannot specify the observation set, the distinct-mechanisms claim lacks statistical support and should be downgraded to exploratory. If the correlations use per-seed or per-task observations instead, report the actual n and verify that the p-values hold under that specification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's second headline claim—that CPT methods with similar downstream performance achieve robustness through distinct mechanisms—rests entirely on the correlational analyses in Figure 2, which relate downstream improvement over BASE to (a) BPC improvement and (b) standard–dialect cosine-similarity increase. The text reports p-values such as r=0.810, p=.003 for CHAR, but never states the number or identity of the observations entering these correlations. Tables 13 and 14, the only mechanistic data shown, contain just three aggregate language–setting rows (German intent, Italian intent, Arabic topic) after macro-averaging over dialects and seeds. With three points, r=0.810 has p approximately 0.40, not p=.003; more generally, none of the reported p-values are consistent with n=3. The values would be roughly consistent with n about 11, which happens to equal the number of dialect varieties in the three parallel settings, suggesting the correlations may have been computed at the per-dialect level—but the paper neither says so nor reports per-dialect BPC/cosine values. Because the 'different mechanisms' claim is the paper's most novel contribution, and because its evidential basis is not reproducible from the manuscript, the claim is currently unsubstantiated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents a systematic comparison of six continued pre-training (CPT) conditions—BASE, CLEAN, BPE-DROP, SUB-REP, CHAR, and PHON—applied to Llama-2-7B and, in a subset, Qwen2-7B, evaluating zero-shot dialect robustness on German, Italian, and Arabic tasks. The authors report that perturbation-based CPT, especially character-level noise (CHAR), consistently improves dialect performance while largely preserving standard-variety accuracy. They further claim that methods with similar downstream performance achieve robustness through distinct mechanisms, evidenced by different correlations between downstream gains and two mechanistic variables: bits-per-character (BPC) improvement and standard–dialect cosine similarity increase. The paper includes auxiliary analyses of prediction repair, tokenization statistics, and cross-linguistic differences.","tokens_in":15194,"tokens_out":8829,"duration_ms":69586,"significance":"If the mechanism claim is adequately supported, the paper would be a valuable contribution to the growing literature on dialect robustness: it moves beyond simple benchmark comparisons to ask how different synthetic perturbations induce robustness, and it offers practical guidance for selecting CPT strategies. Strengths include the controlled comparison of multiple perturbation families on a fixed architecture and training setup, calibration of perturbation strengths to match tokenization statistics, evaluation across three typologically distinct languages and multiple tasks, and a second model family as a generalization check. The prediction-repair analysis provides a useful additional lens. However, the central novelty—the 'different mechanisms' claim—currently rests on correlational evidence whose statistical basis is not transparent or reproducible, which limits the paper's significance until that evidence is properly reported and verified.","major_comments":[{"comment":"The paper claims that '43 of 45 language–task–method comparisons' outperform BASE, but Table 3 does not support this count. A careful count from Table 3 yields 42 comparisons above BASE: there are two ties (German NER CLEAN at 42.7 = 42.7, Italian NER PHON at 51.6 = 51.6) and one below-base result (Arabic NER PHON at 50.1 vs. BASE 50.3). The qualitative conclusion that CPT reliably improves dialect robustness remains intact, but the stated numerical result is inaccurate and must be corrected.","section":"§5.1 and Contribution 2"},{"comment":"The 'distinct mechanisms' claim—the paper's central novelty—relies on Pearson correlations reported in Figure 2, but the manuscript never states the number of observations entering these correlations. Tables 13 and 14 report BPC and cosine similarity for only three language–setting aggregates (German intent, Italian intent, Arabic topic). If the correlations are computed over these three points, the reported p-values (e.g., r=0.810, p=.003 for CHAR) are arithmetically impossible under the standard t-test with n=3 (df=1, p would be approximately 0.40). The p-values are roughly consistent with n≈11 for CHAR and PHON and n≈33 for CLEAN/BPE/SUB-REP, suggesting the analysis might have been performed at the per-dialect or per-seed level, but the paper does not say so, nor does it report the per-dialect BPC/cosine values needed to reproduce the correlations. Without a clear statement of the unit of analysis, the number of observations, and the underlying data, the 'different mechanisms' claim is not reproducible from the manuscript as written.","section":"§5.3, Figure 2, Tables 13 and 14"}],"minor_comments":[{"comment":"The caption refers to 'NOISY' but the corresponding CPT condition is called 'CHAR' throughout the text; please reconcile the terminology.","section":"Table 13 caption"},{"comment":"The column header 'BPE' is used in Table 3 but the method is introduced as 'BPE-DROP' in Section 3.1; use a single consistent abbreviation.","section":"Tables 3 and throughout"},{"comment":"The phrase 'nine German, Italian, and Arabic dialect tasks' is ambiguous: there are three languages and nine tasks total. Reword to 'nine dialect tasks across German, Italian, and Arabic' for clarity.","section":"Abstract"},{"comment":"For each reported correlation, please include the sample size (n) either in the text or in the figure caption, and consider showing the scatter plots so that the relationship is visually verifiable.","section":"§5.3 and Figure 2"},{"comment":"The caption states values are 'macro-averaged over dialect varieties,' but for Italian intent detection there is only one dialect variety (Neapolitan); clarify whether 'macro' is meaningful for that row.","section":"Table 14 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper's main robustness finding is solid and likely publishable after corrections, but the 'distinct mechanisms' claim is currently the paper's headline contribution and its statistical foundation is not adequately reported. The authors should be asked to provide the exact sample sizes, the unit of analysis, and either the per-dialect data or scatter plots for the Figure 2 correlations, and to recompute or justify the reported p-values. The '43 of 45' count error, while easy to fix, should also be corrected before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The practical finding is solid; the mechanism claim is not yet supported. The paper systematically compares six continued pre-training conditions across three languages and nine tasks, calibrates perturbation strength rather than eyeballing it, averages three seeds, and checks standard-variety preservation plus a Qwen replication. Table 3 does support the main takeaway: character-level noise is the strongest default, winning six of nine tasks, though not universal. That's a genuinely useful, reproducible result.\n\nThe soft spots are real. The text says 43 of 45 comparisons outperform BASE, but Table 3 gives 42, with two ties (German NER CLEAN, Italian NER PHON) and one below-base (Arabic NER PHON). That's a small but telling factual error.\n\nThe bigger problem is Figure 2. The 'different mechanisms' claim rests on Pearson correlations between downstream improvement and BPC/cosine-similarity changes. Nowhere does the paper state the sample size. Tables 13 and 14, the only mechanistic data shown, contain three aggregate rows (DE, IT, AR). If those are the data points, r=0.810 with n=3 gives p≈0.40, not p=.003. If the correlations are computed per dialect, the per-dialect BPC and cosine values are never shown. Either way, the central novelty is currently unsubstantiated. The authors need to report the correlation sample, show per-dialect values, or reframe the mechanism story as suggestive.\n\nAlso, the claim that CLEAN is best for LM fit is shaky: on German dialect BPC, CHAR improves more than CLEAN (3.6 to 3.2 vs 3.6 to 3.3). It's not fatal, but it shows the mechanism attribution is less clean than the prose suggests.\n\nOverall, the empirical comparison is worth having, and the practical guidance (CHAR as default) holds up. The mechanism analysis needs a statistical audit before it can be trusted. I'd send this to review—the main experiment is careful and the data are useful—but the authors must fix the count and disclose the correlation sample size. A reader interested in dialect robustness will get value from Tables 3 and 5; the mechanism sections are a cautionary tale about under-reported statistics.","headline":"The practical finding is solid—character-level noise is a strong default for robustness-focused CPT—but the 'distinct mechanisms' claim is statistically underspecified and needs a closer look before it can be trusted.","tokens_in":15691,"tokens_out":3345,"would_cite":true,"duration_ms":28161,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Character-noised continued pre-training reliably improves zero-shot dialect robustness in multilingual LLMs, and similar downstream scores can arise from distinct mechanisms.","keywords":["dialect robustness","continued pre-training","perturbation-based CPT","character-level noise","zero-shot robustness","multilingual language models","representation alignment","prediction repair"],"falsifier":"Recompute Figure 2's correlations using per-seed or per-dialect observations rather than three seed-averaged settings and test whether CHAR's downstream gains still track standard–dialect cosine similarity while CLEAN's track bits per character; the paper reports p-values but never states the sample size, and Tables 13 and 14 report the mechanistic variables for only three parallel settings. A behavioral control would run CHAR on a dialect pair whose divergence is lexical rather than orthographic: the mechanism account predicts a smaller gain.","tokens_in":14723,"feed_emoji":"🗣️","tokens_out":12116,"duration_ms":95402,"temperature":0.7,"pith_summary":"This paper argues that perturbation-based continued pre-training (CPT) is a dependable way to make multilingual LLMs more robust to dialects they have never been trained on, and that different perturbation recipes achieve that robustness through different internal routes. Across nine German, Italian, and Arabic tasks and six training conditions, 43 of 45 dialect evaluations improve over the base model, with an average gain of 3.0 points on dialect data and a small net gain on standard varieties. Character-level noise is the strongest single strategy, but the more general point is that similar benchmark numbers can hide different mechanisms: clean and token-level perturbations mostly improve language-model fit, while character-level and phonological perturbations mostly increase the similarity between standard and dialect representations. The paper therefore reframes perturbation-based CPT as a family of adaptation strategies, not a single intervention, and gives practitioners a principled basis for choosing among them.","feed_headline":"Character noise gives LLMs the strongest dialect robustness gains","feed_subtitle":"Character noise repairs the most dialect errors while keeping standard accuracy; equal scores hide different mechanisms.","key_machinery":"The machinery is a controlled comparison of six CPT conditions with calibrated perturbation strength, plus two mechanistic probes. Bits per character (BPC) measures how well each checkpoint predicts natural text; standard–dialect cosine similarity measures how similarly the model represents paired standard and dialect sentences. The paper also tracks prediction transitions between BASE and each CPT model, defining the net repair rate as corrected BASE errors minus newly introduced errors, and correlates each mechanistic variable with downstream dialect gains to attribute credit. This combination separates perturbation type from perturbation severity and links benchmark improvements to specific behavioral and representational changes.","core_discovery":"The central claim is that perturbation-based continued pre-training, especially character-noised CPT, reliably improves zero-shot dialect robustness in decoder-only LLMs while largely preserving standard-variety performance, and that methods with nearly equal downstream accuracy can be robust for different reasons. The authors compare BASE, CLEAN, BPE dropout, subword replacement, character noise, and a phonological noiser on German, Italian, and Arabic, training linear probes on frozen representations to isolate the effect of CPT. CHAR achieves the best dialect-average score in six of nine language–task settings and the highest net repair rates, while CLEAN, BPE-DROP, and SUB-REP correlate with improved language-model fit and CHAR and PHON correlate with increased standard–dialect cosine similarity. They conclude that robustness from synthetic surface variation is not one phenomenon, and that languages with larger standard–dialect tokenization gaps, such as German and Italian, offer more room for these interventions than Arabic.","pith_inferences":["My inference, not the paper's: pairing a fit-improving CPT (clean or token-level) with an alignment-improving CPT (character or phonological noise) may combine the two mechanisms; the paper does not test combined strategies.","My inference: because all downstream evaluations freeze the LLM and train linear probes, the mechanism split describes representation quality for discriminative tasks; the paper itself limits its conclusions to those settings, so open-ended generation remains an untested extension.","My inference: the orthographic-divergence account predicts character noise should transfer best to other spelling-heavy dialect pairs, for example Scots or Latin-script dialectal Arabic, and worse where dialect divergence is mostly lexical or syntactic."],"forward_implications":["Character-level noise is the strongest general-purpose CPT choice for zero-shot dialect robustness, winning six of nine language–task settings in this study.","Similar benchmark scores are not evidence of the same intervention effect: clean and token-level CPT correlate with better language-model fit, while character and phonological CPT correlate with higher standard–dialect representation similarity.","CPT gains come primarily from repairing baseline errors rather than introducing new ones, with CHAR showing the largest net repair rates.","Languages with larger standard–dialect tokenization gaps, such as German and Italian, offer more room for perturbation-based CPT than Arabic, whose dialect surface forms are closer to the standard."],"supporting_citations":[{"why":"Supplies the decoder-only noisy continued pre-training recipe (LoRA, 0.5 keep probability, checkpointing) that the paper adapts and extends to three languages.","marker":"Kojima et al., 2025"},{"why":"Source of the character-level noise operator used for CHAR, originally developed for dialect and noisy text.","marker":"Srivastava and Chiang, 2023a"},{"why":"Earlier demonstration that character-level noise improves zero-shot transfer between closely related languages, motivating the central intervention.","marker":"Aepli and Sennrich, 2022"},{"why":"Provides the phonological noiser used for PHON, modelling systematic standard-to-dialect sound correspondences without dialect resources.","marker":"Bafna et al., 2024, 2025"},{"why":"Defines DialectBench, the source of the Arabic task categorization and several datasets used in the evaluation.","marker":"Faisal et al., 2024"},{"why":"Supplies the xSID4LR intent and slot-filling datasets used for German and Italian downstream evaluation.","marker":"van der Goot et al., 2021"},{"why":"Provides multilingual C4, the corpus whose German, Italian, and Arabic portions are used for continued pre-training.","marker":"Raffel et al., 2020"},{"why":"Provides LoRA, the parameter-efficient fine-tuning method used for all CPT runs.","marker":"Hu et al., 2021"},{"why":"Identifies continued pre-training as a stage of dialect adaptation pipelines, framing why perturbation-based CPT matters.","marker":"Painter et al., 2026"}],"fun_headline_variants":["Character noise tops dialect robustness in continued pretraining","Equal scores, different fixes: How perturbations reshape dialect robustness","Perturbation recipes: Character noise wins for zero-shot dialect tasks","Why character noise fixes dialect errors: Different paths to same score"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The distinct-mechanisms conclusion depends on correlations computed over only three language settings (German intent, Italian intent, Arabic topic); with so few data points, a different selection of dialects could change which mechanism looks responsible.","fun_headline_variants_meta":{"raw":{"variants":["Character noise tops dialect robustness in continued pretraining","Equal scores, different fixes: How perturbations reshape dialect robustness","Perturbation recipes: Character noise wins for zero-shot dialect tasks","Why character noise fixes dialect errors: Different paths to same score"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000571,"raw_usage":{"total_tokens":2667,"prompt_tokens":879,"completion_tokens":1788,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":495,"completion_tokens_details":{"reasoning_tokens":1719}},"tokens_in":495,"tokens_out":1788,"duration_ms":12043,"temperature":1.0,"reasoning_tokens":1719,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T11:45:56.946870+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute Figure 2's correlations using per-seed or per-dialect observations rather than three seed-averaged settings and test whether CHAR's downstream gains still track standard–dialect cosine similarity while CLEAN's track bits per character; the paper reports p-values but never states the sample size, and Tables 13 and 14 report the mechanistic variables for only three parallel settings. A behavioral control would run CHAR on a dialect pair whose divergence is lexical rather than orthographic: the mechanism account predicts a smaller gain.","supporting_citations":[{"cited_title":"2024 , url =","cited_arxiv_id":null,"evidence_quote":"Defines DialectBench, the source of the Arabic task categorization and several datasets used in the evaluation."}],"review_version":1}