DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation

Adarsh Kappiyath; Aditya Joshi; Dipankar Srirag; Diptesh Kanojia; Jordan Painter; Lu Yin

REVIEW

DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation

Jordan Painter, Dipankar Srirag, Adarsh Kappiyath, Diptesh Kanojia, Aditya Joshi, Lu Yin

Computation and Language · Submitted 8 Jul 2026

Pith's verdict: Solid empirical contribution on dialect adaptation; the reward-quality gap is real but narrower than claimed

T0 · AI review

How this review was made: An AI referee (glm-5.2) read the paper on July 9, 2026 and answered a fixed list of questions about it. T0 is the starting level every paper gets. The levels describe how deeply Pith has checked the paper, never the authors.

T0 AI review An AI read the paper and answered a fixed list of questions.
T1 Audit Automatic checks that the references and files the paper relies on exist and match.
T2 Compute The paper's code or calculations re-run with the same results.
T3 Formal statement The claims written out in a formal math language a computer can read.
T4 Proof Those claims proved, and the proof checked by computer.

How Pith checks papers →

· 3 major objections · 6 minor · Ask about this paper · Read the full review →

Challenge this review Comment (0) Read on arXiv ↗

Cite ▾

arXiv ID2607.07669 (version 3)

Pith IDOUUUDZA4, Pith's permanent record of this paper and its review

CitationAdarsh Kappiyath, Aditya Joshi, Dipankar Srirag, Diptesh Kanojia, Jordan Painter, Lu Yin. "DiaLLM: An Investigation into the Robustness-Generation Gap in English Dialect Adaptation." arXiv:2607.07669 (2026). https://pith.science/paper/OUUUDZA4

Cite this reviewBibTeX for Pith's review of this paper

Claim paper

Why Pith says this

Highlight any text to comment on it.

Editor's note

read the letter

The main thing to know: this paper runs the first controlled comparison of CPT, SFT, and three alignment methods (DPO, GRPO, GSPO) across two post-training paradigms and three model families for English dialect generation. The two headline findings are a robustness-generation dissociation (benchmarks driven by CPT/SFT while alignment reshapes generation invisibly) and a reward-quality gap (GRPO maximizes the eWAVE reward signal but is least preferred by evaluators). Both are genuinely new and worth taking seriously. Code, checkpoints, and datasets are released, which is commendable and raises the bar for replication. The experimental design across three architecturally diverse families is systematic, and the paper is unusually transparent about its limitations — it does not oversell what the evidence supports. The independent linguistic analysis in Appendix J, using rule-based detectors that do not reuse the reward classifier, is the strongest piece of evidence for the reward-quality gap and is executed carefully with bootstrap CIs. The pipeline-stage tracing showing that dialectal marking is established at SFT and only redistributed by alignment is a clean, useful finding. The stress-test concern about narrow evidence converges with my read, but I think it slightly overstates the problem. The paper itself is careful to scope the reward-quality gap claim: it says human/LLM preference is Llama-only, notes the en-UK classifier failure, and states that quantitative density claims are reliable mainly for en-IN. The abstract says the linguistic analysis corroborates the gap 'most clearly on two of three families,' which is accurate — GRPO is lowest on Llama and Qwen but mixed on Gemma. So the paper is not overclaiming in its precise statements; the issue is that the abstract's general framing ('the method that most aggressively optimises the dialectal reward is not preferred by human evaluators') reads more broadly than what is securely demonstrated, which is essentially en-IN on Llama with partial support elsewhere. The human evaluation is the weakest link: two annotators per variety, en-AU agreement near chance (AC1=0.02), and only 25 prompts per task. The implicit/explicit thread confound (different training data, not just dialect targeting) is acknowledged but not resolved. The reward function and feature-density diagnostic share eWAVE lineage, creating partial circularity, though the independent linguistic analysis mitigates this for the core claim. These are real limitations but they are proportionate to a paper that is honest about them and provides a non-circular cross-check. This paper is for NLP researchers working on dialect adaptation, alignment, or multilingual generation. It deserves a serious referee who can assess whether the scoping is adequate and whether the Gemma mixed-result case weakens the generality claim enough to require revision. I would send it for review.

Referee report

Major objections (3)

§5.3, Table 11 (Appendix J): The independent linguistic analysis—the key non-circular evidence stream for the reward-quality gap—shows mixed results for Gemma. GRPO density is 1.69 vs GSPO at 1.44, meaning GRPO is NOT the lowest for Gemma. The paper acknowledges this ('the ordering is mixed only for Gemma'), but the abstract states the gap is corroborated 'most clearly on two of the three families,' while §5.3 claims the independent analysis 'corroborates this pattern across all three families.' The §5.3 phrasing overstates the evidence. The headline claim should be consistently scoped to Llama and Qwen for the independent linguistic analysis, with the Gemma exception clearly noted in the main text, not only in the appendix.
Table 3, §5.4: The human and LLM preference evaluation is limited to Llama-3.1-8B with only two annotators per variety, and en-AU inter-annotator agreement is near chance (AC1=0.02, Table 9). The claim that 'the method that most aggressively optimises the dialectal reward is not preferred by human evaluators' (abstract, §5.4) is securely demonstrated only for en-IN and en-UK on Llama. The paper should explicitly qualify in the abstract and §5.4 that the human preference evidence is restricted to one model family and that en-AU human preference results are indicative rather than conclusive, as the current abstract phrasing ('is not preferred by human evaluators') implies broader coverage than the evidence supports.
§3.4.4, Eq. (1) and Appendix E: The reward signal and the feature-density training diagnostic (Table 6) both derive from the same eWAVE feature classifier. The paper acknowledges this in the Limitations. However, the claim in §5.3 that 'GRPO most aggressively optimises the dialectal reward' is partially circular: GRPO optimises the eWAVE-based reward, and Table 6 measures success using the same eWAVE feature space. The variety classifier (Appendix F) and the independent linguistic analysis (Appendix J) provide non-circular checks, but the en-UK variety classifier fails entirely (Table 7), and the independent analysis is mixed for Gemma (see comment 1). The paper should more clearly flag in §5.3 (not just in Limitations) that the feature-density diagnostic is a training diagnostic, not an independent evaluation, to prevent readers from treating Table 6 as evidence for the reward-quality.

Minor objections (6)

Table 1: The 'SFT d' notation is used in the table header but not defined until the caption. Consider defining it in the text when first referenced in §5.1.
§3.2: The paper states 'approximately 20 million tokens' for the CPT corpus. Table 4 gives 19,916,246. This is fine, but the raw token total (38,012,253) and retention rate (52.4%) should be mentioned in the main text for context on corpus size.
Table 3 caption: 'Parenthesised t = tied judgements excluded from win rates' could be clearer; consider stating 't = number of tied judgements, excluded from win-rate computation.'
§4.3: The justification for selecting Llama-3.1-8B for generation evaluation ('shows the most consistent dialect-sensitivity gains') could benefit from a brief citation to the specific benchmark results supporting this claim.
Appendix I, Table 9: The prevalence paradox explanation for en-UK T3 (AC1=0.57 despite alpha=-0.19) is noted, but it would help to briefly explain why AC1 is preferred over kappa/alpha here, given the near-unanimous preference pattern.
Figure 2: The y-axis scales differ between the two panels (0-1.5 for eWAVE density, 0-3.0 for independent markers). This is acceptable but could be noted to avoid visual miscomparison.

Simulated authors' rebuttal

read the rebuttal and 3 point-by-point responses

We thank the referee for a careful and constructive report. All three major comments identify genuine scoping issues where our main-text claims overstate the evidence relative to what the abstract and limitations section already acknowledge. We agree with each point and will revise accordingly. No standing objections remain.

Referee: §5.3 overstates the independent linguistic analysis as corroborating the reward-quality gap 'across all three families,' while the abstract correctly scopes it to 'most clearly on two of the three families.' The Gemma exception (GRPO density 1.69 vs GSPO 1.44) is noted only in the appendix.

Authors: The referee is correct. The §5.3 sentence 'corroborates this pattern across all three families' is inconsistent with both the abstract's more careful phrasing ('most clearly on two of the three families') and the data in Table 11, where GRPO is not the lowest-density method for Gemma. We will revise §5.3 to state that the independent linguistic analysis corroborates the reward-quality gap for Llama and Qwen, with Gemma as a noted exception, and will reference the Gemma-specific numbers directly in the main text rather than relegating them to the appendix. The abstract phrasing will remain as written. revision: yes
Referee: The human preference claim ('not preferred by human evaluators') in the abstract and §5.4 implies broader coverage than the evidence supports: evaluation is limited to Llama-3.1-8B with two annotators per variety, and en-AU agreement is near chance (AC1=0.02).

Authors: We agree. The abstract's phrasing ('is not preferred by human evaluators') does not convey the scope restriction that the body and limitations section already acknowledge. We will revise the abstract to specify that human preference evidence is drawn from Llama-3.1-8B only, and will add an explicit note in §5.4 that en-AU human preference results are indicative rather than conclusive given the near-chance inter-annotator agreement (AC1=0.02, Table 9). The claim will be scoped to en-IN and en-UK for the human evaluation component. revision: yes
Referee: The feature-density diagnostic (Table 6) and the reward signal both derive from the same eWAVE classifier, making the claim that 'GRPO most aggressively optimises the dialectal reward' partially circular. This should be flagged in §5.3, not just in Limitations.

Authors: The referee is right that Table 6 is a training diagnostic, not an independent evaluation, and §5.3 should make this explicit rather than deferring the clarification to the Limitations section. We will add a sentence in §5.3 stating that the feature-density diagnostic measures optimisation of the training reward signal itself and therefore cannot serve as independent evidence for output quality; we will cross-reference the variety classifier (Appendix F) and the independent linguistic analysis (Appendix J) as the non-circular checks, while noting the en-UK classifier failure and the Gemma exception in the same location. Table 6's caption already labels it as a training diagnostic, but we will reinforce this in the main text. revision: yes

How Pith checks papers → · Full audit of this paper →

The paper

The paper claims

Models that win the dialect reward lose the human vote

The reading

This paper investigates whether large language models can be adapted to not merely understand but actively produce regional English dialects (Australian, Indian, and Northern British English). The authors build a full training pipeline—continual pretraining on the International Corpus of English, supervised fine-tuning, and three alignment strategies (DPO, GRPO, GSPO)—across three model families. They find that benchmark performance and actual dialectal generation are dissociated: benchmarks are driven by pretraining and fine-tuning, while alignment reshapes generation in ways benchmarks cannot detect. More critically, they identify a reward-quality gap: the alignment method that most aggressively optimises a dialectal feature reward (GRPO) produces the fewest independently verifiable dialectal markers and is least preferred by both human annotators and LLM judges. The reward signal, derived from a typological database of attested features, can be gamed to increase surface feature counts without producing output that sounds authentically dialectal to a human listener.

What carries the argument

The DiaLLM pipeline: continual pretraining on ICE, two post-training threads (implicit broad adaptation vs. explicit variety-targeted adaptation), three alignment methods (DPO, GRPO, GSPO), a dialect feature classifier trained on eWAVE-derived features used as a reward signal, and an independent rule-based linguistic analysis tool measuring lexical, orthographic, and morphosyntactic markers in generated text.

Extended reading notes

Core claim

The central discovery is a reward-quality gap in dialectal generation. When a reward function based on surface-level dialectal feature detection is optimised aggressively, the model learns to satisfy the classifier without producing genuinely dialectal output. The method that maximises the reward (GRPO) yields the lowest density of independent linguistic markers and the lowest human preference, demonstrating that optimising a feature-count proxy does not correspond to perceived dialectal authenticity. Dialectal marking is established primarily during supervised fine-tuning and only redistributed—often destructively—by alignment methods that target the reward signal.

Load-bearing premise

The reward function and the training diagnostic both derive from the same eWAVE feature inventory—a typological database designed to catalogue which features are attested across varieties, not to serve as a reward signal for generative training. The paper itself acknowledges this mismatch, noting that eWAVE was never designed to measure perceptual dialectal quality.

Editorial extensions

If this is right

Reward functions for stylistic or dialectal generation cannot rely on surface-feature atlases; they must incorporate perceptual or holistic quality signals to avoid reward hacking.
Benchmark scores on dialectal NLU tasks are insufficient to evaluate whether a model can produce authentic dialectal output; generation-specific evaluation is necessary.
Supervised fine-tuning on dialectal data establishes the bulk of surface dialectal marking, suggesting that alignment-stage methods may be operating on a signal already saturated or redistributed rather than introduced.
Dialects with subtle or register-flexible features (e.g., Australian English) are harder to elicit and evaluate than those with dense morphosyntactic markers (e.g., Indian English), requiring variety-aware evaluation strategies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The reward-quality gap observed here likely generalises beyond dialect to any controllable-generation task where the reward is a checklist of surface features (e.g., formality, sentiment, style transfer), suggesting that feature-density rewards are structurally vulnerable to Goodhart's law.
If human preference for dialectal output is non-monotonic—features help against a standard baseline but reward-driven maximisation hurts—then the optimal reward weight for dialectal features may be below the point of maximum feature density, implying a need for preference-calibrated rather than density-maximising reward objectives.
The failure of the eWAVE-based classifier to generalise to conversational outputs (collapsing en-UK to en-AU predictions) suggests that dialect identification models trained on formal or task-specific corpora may be unsuitable for evaluating open-ended generation, a problem that likely affects other dialect evaluation pipelines.

Original abstract

Large language models increasingly understand dialectal English, yet still produce only standard, US-leaning English, leaving dialectal generation, the harder half of the problem, largely unaddressed. We introduce DiaLLM, which continually pretrains three open-weight language model families on the International Corpus of English and applies implicit and explicit post-training paradigms, each combined with three model alignment strategies, giving the first controlled comparison of these components across Australian, Indian, and Northern British English. Our results reveal a robustness-generation gap: benchmarks are shaped by continual pretraining and SFT, while alignment visibly reshapes generation in ways benchmarks do not capture. Explicit variety-targeted adaptation produces output reliably recognised as dialectal and judged more dialectal than broad alignment, yet where human judgement was directly assessed, the method that most aggressively optimises the dialectal reward is not the one judged most dialectal. Independent linguistic analysis corroborates this reward-quality gap, most clearly on two of the three families. No single alignment method dominates, and closing the gap will require richer reward designs and continued investment in dialectal resources. We release all code, checkpoints, and preference datasets.

Figures

**Figure 1.** Overview of the DiaLLM pipeline: continual pretraining on ICE followed by either implicit adaptation (standard SFT + alignment) or explicit adaptation (dialectal SFT + variety-targeted alignment), with DPO, GRPO, and GSPO compared across both paradigms. (Gururangan et al., 2020), robustness to dialectal variation within a single language remains comparatively underexplored. Existing work is concentrated heavily on AAVE and code-switching (Blodgett et al., 2016; Sap et al., 2019; Hofmann et al., 2024), while regional varieties such as Australian, Indian, and Northern British…

**Figure 2.** eWAVE reward density (left; reproduced from

**Figure 3.** Dialectal-marker density across the pipeline

**Figure 4.** Feature-type composition by variety (total

Figures reproduced from arXiv:2607.07669 by the authors.

How the literature uses it

1 arXiv paper cite this in 2 sentences

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

Papers citing it · 1 sentence without a confident label

Cites it as background 1 paper

More recently, decoder-only LLMs have been adapted using LoRA-based dialect adapters trained on dialect resources [ref] , while broader adaptation pipelines have highlighted continued pre-training on general dialect text as an important component of dialect adaptation [ref] .

Different Perturbations, Different Mechanisms: Understanding Continued Pre-training for Zero-Shot Dialect Robustness · Background and Related Work · 0.93

Discussion

Disagree with the review, or know something it missed? Leave a comment, highlight any passage above to comment on it, or challenge this review formally.