pith. machine review for the scientific record. sign in

arxiv: 2601.22888 · v3 · submitted 2026-01-30 · 💻 cs.CL · cs.AI

Recognition: 2 theorem links

· Lean Theorem

DialectLLM: A Dialect-Aware Dialog[ue] Generation Framework Beyond Standard American English

Jio Oh , Paul Vicinanza , Thomas Butler , Steven Euijong Whang , Dezhi Hong , Amani Namboori
Authors on Pith no claims yet

Pith reviewed 2026-05-16 09:50 UTC · model grok-4.3

classification 💻 cs.CL cs.AI
keywords dialect generationdialogue systemsLLM evaluationEnglish dialectsnatural language generationpost-trainingmultidialectal NLP
0
0 comments X

The pith

A framework generates authentic multi-dialect conversational data for nine English variants by applying linguist-validated rules to Standard American English dialogs.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

More than 80 percent of English speakers use dialects other than Standard American English, yet large language models often misidentify them and generate stereotyped or inaccurate responses. DialectLLM creates parallel dialog datasets across nine dialects by converting SAE conversations using separate rules for lexical vocabulary, orthographic spelling, and morphosyntactic grammar features, with rules validated by native linguists. Human raters prefer the generated data for naturalness in 98.8 percent of pairwise comparisons against prior methods. The resulting DialectLLM-Bench of over 50,000 dialogs exposes that even frontier models reach under 70 percent accuracy on dialect identification and under 50 percent on prominent dialects like Canadian English, while systematically labeling non-SAE speech as American or British. The same data also functions as scalable material for post-training to produce more dialect-aware conversational models.

Core claim

DialectLLM is the first large-scale framework that produces a dialect-parallel dialog dataset spanning nine English dialects through SAE-to-dialect transformation rules designed and validated with native linguists to cover lexical, orthographic, and morphosyntactic features, while demonstrating that models should avoid reproducing up to 90 percent of a dialect's grammatical features in their responses.

What carries the argument

SAE-to-dialect transformation rules that handle lexical, orthographic, and morphosyntactic features separately for user utterances versus model responses and are validated by native linguists for authenticity.

If this is right

  • Frontier models achieve under 70 percent accuracy on dialect identification and under 50 percent for dialects such as Canadian English.
  • Non-SAE dialects are systematically misclassified as American or British English by current models.
  • The generated data serves as a scalable post-training resource to improve dialect-aware conversational performance.
  • Models should not reproduce up to 90 percent of the grammatical features when generating responses in a given dialect.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Wider adoption of separate feature rules for responses could reduce stereotypical outputs in global English conversations.
  • The parallel dataset structure may support targeted fine-tuning experiments to close specific dialect gaps.
  • Similar rule-based transformation approaches could be tested on spoken rather than written dialect data.
  • Benchmark results suggest value in expanding evaluation to real-time multi-turn interactions with dialect speakers.

Load-bearing premise

The SAE-to-dialect transformation rules designed and validated with native linguists accurately capture authentic dialect features without introducing new stereotypes or artifacts that human raters would miss.

What would settle it

Native speakers rating DialectLLM-generated dialogs as less natural than real dialect speech or than outputs from earlier methods, or models fine-tuned on the data showing no measurable gains on dialect identification or generation tasks.

Figures

Figures reproduced from arXiv: 2601.22888 by Amani Namboori, Dezhi Hong, Jio Oh, Paul Vicinanza, Steven Euijong Whang, Thomas Butler.

Figure 1
Figure 1. Figure 1: Overview of MDial. The framework combines ortho-lexical seed creation, multi-turn SAE dialog generation, and rule-guided dialect transformation using curated lexical, orthographic, and morphosyntactic knowledge to produce parallel dialogs across 9 English dialects. The resulting dataset supports both benchmarking (MDialBench) and post-training to enhance LLMs’ dialectal capabilities. Parametric transformat… view at source ↗
Figure 2
Figure 2. Figure 2: Annotation rating distribution of eWAVE, User, and Model appropriate generation for AU morphosyntax features. should not be produced by the model ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Model performance on MDialBench classification (Iden￾tification) task for turn 1 and 8 dialogs. We observe an overall trend of positive correlation between model size & accuracy and that models struggle when morphosyntactic features are added. Results across all turns for classification and generation are shown in [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Model performance by scale and context length averaged over OrthoLex, RBTUser, and RBTmodel data. we also evaluate a model trained on just 0.15 epochs (a couple hundred examples per dialect) and find performance still exceeds frontier models. The finite state space for dialect variations—lexical, orthographic, and morphosyn￾tactic mappings—means that our training data enable the model to learn comprehensiv… view at source ↗
Figure 5
Figure 5. Figure 5: Guidelines provided to Nigerian annotators for evaluating morphosyntactic variations. Examples are truncated for brevity. 12 [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Guidelines provided to Irish annotators for evaluating lexical and orthographic variations. Examples are truncated for brevity. B. Details on Data Generation B.1. Dialectal Transformation Features [PITH_FULL_IMAGE:figures/full_fig_p013_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: LLM-based wordbank matrix completion. Using human annotations as few-shot demonstrations (diagonal yellow), we prompt an LLM to fill out missing entries in the dialectal mapping matrix (blue regions). B.7. Annotation Rating Alignment and Distribution (eWAVE vs MDial) Continuing from Sec. 3.1, we report the feature-level alignment rate ( [PITH_FULL_IMAGE:figures/full_fig_p018_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Annotation rating distribution of eWAVE, F eaturesUser, and F eaturesModel for dialects that exist in the eWAVE database. We observe clear discrepancies between the new ratings from native linguists and the original eWAVE annotations, with larger differences for F eaturesModel. 19 [PITH_FULL_IMAGE:figures/full_fig_p019_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: LLMaaJ evaluation results for open-ended generation. Scores across orthographic, lexical, and morphosyntactic judges confirm that performance scales with model size, consistent with other tasks. Moreover, we can observe that LLMs are less capable of incorporating appropriate morphosyntactic features compared to lexical or orthographic features. 22 [PITH_FULL_IMAGE:figures/full_fig_p022_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Model performance on MDialBench classification (Identification) task. We observe an overall trend of positive correlation between model size & accuracy and that models struggle when morphosyntactic features are added. 23 [PITH_FULL_IMAGE:figures/full_fig_p023_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Model performance on MDialBench generation (response completion) task. Similar trend to [PITH_FULL_IMAGE:figures/full_fig_p024_11.png] view at source ↗
Figure 12
Figure 12. Figure 12: Scaling behavior of LLMs on dialectal classification and generation tasks. We show the model performance at turn 8, which has the most contextual cues and has the least multi-labels, to avoid giving credit for models outputting US or GB-biased answers. Accuracy scales linearly with model size. D.3. More Results from Sec. 5.5 In this section, we show the confusion matrices of different models. More context… view at source ↗
Figure 13
Figure 13. Figure 13: Confusion matrix for Qwen3-1.7B (left) and Claude-Opus-4.1 (right) on 8-turn dialogs; classification task. For Qwen3-1.7B, The vertical clustering in the US and GB columns suggests that the model is making default guesses to standard dialects rather than analyzing dialectal features. On the other hand, for Claude-Opus-4.1, the diagonal pattern indicates that the model is capable of genuinely identifying d… view at source ↗
Figure 14
Figure 14. Figure 14: Confusion matrices of different models averaged over all turns (1, 2, 4, and 8); classification task. Smaller models show more bias towards high-resource, standard dialects (US, GB). 26 [PITH_FULL_IMAGE:figures/full_fig_p026_14.png] view at source ↗
Figure 15
Figure 15. Figure 15: Confusion matrices of different models averaged over all turns (1, 2, 4, and 8); classification task. Smaller models show more bias towards high-resource, standard dialects (US, GB). 27 [PITH_FULL_IMAGE:figures/full_fig_p027_15.png] view at source ↗
Figure 16
Figure 16. Figure 16: Impact of test-time compute on classification (left) and generation (right) task. Models equipped with reasoning capabilities (dotted lines) achieve a consistent performance increase over their base counterparts. 28 [PITH_FULL_IMAGE:figures/full_fig_p028_16.png] view at source ↗
Figure 17
Figure 17. Figure 17: Prompt for natural multi-turn seed dialog generation. 30 [PITH_FULL_IMAGE:figures/full_fig_p030_17.png] view at source ↗
Figure 18
Figure 18. Figure 18: Prompt for indirect multi-turn seed dialog generation. 31 [PITH_FULL_IMAGE:figures/full_fig_p031_18.png] view at source ↗
Figure 19
Figure 19. Figure 19: Prompt template for the construction of the OrthoLexinitial dataset. The italicized sentence is included only when the mapping originates from the target transformation wordbank. 32 [PITH_FULL_IMAGE:figures/full_fig_p032_19.png] view at source ↗
Figure 20
Figure 20. Figure 20: Prompt for the construction of RBT datasets. 33 [PITH_FULL_IMAGE:figures/full_fig_p033_20.png] view at source ↗
Figure 21
Figure 21. Figure 21: Prompt for filling out the missing words in the wordbank aggregation step. (See [PITH_FULL_IMAGE:figures/full_fig_p034_21.png] view at source ↗
Figure 22
Figure 22. Figure 22: Prompt for filling out the missing ratings in the wordbank aggregation step. (See [PITH_FULL_IMAGE:figures/full_fig_p035_22.png] view at source ↗
Figure 23
Figure 23. Figure 23: An exemplar guideline for the OrthoLex refinement step for Australian English. We provide the model with general spelling conventions for all transformations, while specific changes are provided through probabilistic sampling when the terms appear in the dialogue (“aubergine”, “yes”, “tire” in this example). These transformations are presented in a tabular format, as LLMs demonstrate superior performance … view at source ↗
Figure 24
Figure 24. Figure 24: Prompt for dialect identification. Note that the answer options are shuffled for each instance and when testing the OrthoLex dataset, we remove the mention of “morphosyntactic” for the first instruction. 36 [PITH_FULL_IMAGE:figures/full_fig_p036_24.png] view at source ↗
Figure 25
Figure 25. Figure 25: Prompt for response completion. Note that the answer options are shuffled for each instance and when testing the OrthoLex dataset, we remove the mention of “morphosyntactic” features. 37 [PITH_FULL_IMAGE:figures/full_fig_p037_25.png] view at source ↗
Figure 26
Figure 26. Figure 26: Prompt for LLMaaJ judge that evaluates lexical feature appropriateness. 38 [PITH_FULL_IMAGE:figures/full_fig_p038_26.png] view at source ↗
read the original abstract

More than 80% of the 1.6B English speakers do not use Standard American English (SAE), yet LLMs often fail to correctly identify non-SAE dialects and generate stereotyped responses for their speakers. We introduce DialectLLM, the first large-scale framework for generating high-quality multi-dialectal conversational data encompassing the three pillars of written dialect -- lexical (vocabulary), orthographic (spelling), and morphosyntactic (grammar) features. DialectLLM produces a dialect-parallel dialog dataset spanning nine English dialects. Partnering with native linguists, we design and validate SAE-to-dialect transformation rules, ensuring authenticity. Our approach challenges the prevailing practice of applying a single morphosyntactic feature set to both user utterances and model responses, showing that models should not reproduce up to 90% of the grammatical features of a dialect. Human evaluation confirms data quality, with annotators preferring DialectLLM over prior methods in 98.8% of pairwise comparisons for dialect naturalness. We then construct DialectLLM-Bench, a dialect-parallel benchmark with 50k+ dialogs, resulting in 97k+ QA pairs, and evaluate 17 LLMs on dialect identification and response generation tasks. Even frontier models achieve under 70% accuracy, fail to reach 50% for prominent dialects like Canadian English, and systematically misclassify non-SAE dialects as American or British. Beyond benchmarking, we show that DialectLLM data also serve as a scalable LLM post-training resource, suggesting a practical path toward dialect-aware conversational AI.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces DialectLLM, a framework for generating high-quality multi-dialectal dialog data across nine English dialects using SAE-to-dialect transformation rules designed and validated with native linguists. It reports a human evaluation with 98.8% annotator preference for naturalness over prior methods, constructs the DialectLLM-Bench benchmark containing over 50k dialogs and 97k QA pairs, evaluates 17 LLMs showing poor performance on dialect identification and generation (under 70% accuracy, below 50% for some dialects), and demonstrates the data's value as a post-training resource.

Significance. If the results hold, the work is significant for advancing dialect-aware AI, given that over 80% of English speakers use non-SAE dialects. The linguist-validated rules, extensive human preference study, and new benchmark provide concrete resources and evidence of current LLM limitations. The approach of separating user and response grammar features is a useful insight. The availability of the dataset for post-training is a practical contribution.

major comments (2)
  1. [Human Evaluation] Human Evaluation section: The reported 98.8% preference rate in pairwise comparisons for dialect naturalness lacks error bars, inter-annotator agreement statistics, details on the number of annotators, or their selection criteria, which are necessary to assess the reliability of the human evaluation results supporting data quality.
  2. [Abstract] Abstract: The statement that models should not reproduce up to 90% of the grammatical features of a dialect lacks a clear derivation or empirical basis for the specific 90% threshold; this appears central to the framework's design but requires more justification to support the challenge to prevailing practices.
minor comments (2)
  1. [Title] Title: The title includes 'Dialog[ue]', which appears to be a formatting artifact and should be corrected to 'Dialogue' for consistency.
  2. [Abstract] Abstract: The abstract mentions 'three pillars of written dialect' but could briefly list the nine dialects covered to provide immediate context.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We appreciate the referee's positive recommendation for minor revision and constructive feedback on our manuscript. We address each major comment below and have revised the manuscript to incorporate the suggested improvements.

read point-by-point responses

  1. Referee: [Human Evaluation] Human Evaluation section: The reported 98.8% preference rate in pairwise comparisons for dialect naturalness lacks error bars, inter-annotator agreement statistics, details on the number of annotators, or their selection criteria, which are necessary to assess the reliability of the human evaluation results supporting data quality.

    Authors: We agree that these details are important for assessing reliability. The original manuscript reported the aggregate preference rate but did not include the supporting statistics. We have revised the Human Evaluation section to add the number of annotators, their selection criteria, inter-annotator agreement statistics, and error bars on the reported preference rate. revision: yes

  2. Referee: [Abstract] Abstract: The statement that models should not reproduce up to 90% of the grammatical features of a dialect lacks a clear derivation or empirical basis for the specific 90% threshold; this appears central to the framework's design but requires more justification to support the challenge to prevailing practices.

    Authors: The 90% threshold is derived from the linguist-validated experiments described in Section 3, where applying 90% or more of the morphosyntactic features to model responses was rated as less natural. To address the comment, we have revised the abstract to briefly note this empirical basis and expanded the justification in the main text with reference to the specific validation results. revision: yes

Circularity Check

0 steps flagged

No significant circularity in derivation chain

full rationale

The paper's core contributions rest on externally validated SAE-to-dialect transformation rules developed with native linguists, large-scale human preference evaluations (98.8% win rate), and construction of a new falsifiable benchmark (DialectLLM-Bench). No equations, fitted parameters presented as predictions, or self-citation chains reduce any claim to its own inputs by construction. The argument structure relies on independent empirical evidence rather than self-referential definitions or renamings.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Core claims rest on the validity of linguist-designed transformation rules and the reliability of human naturalness judgments; no free parameters or new entities are introduced.

axioms (1)
  • domain assumption Human annotators can reliably distinguish authentic dialect naturalness from generated text.
    Used to claim 98.8% preference over prior methods.

pith-pipeline@v0.9.0 · 5605 in / 1310 out tokens · 40898 ms · 2026-05-16T09:50:43.358828+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Lean theorems connected to this paper

Citations machine-checked in the Pith Canon. Every link opens the source theorem in the public Lean library.

What do these tags mean?
matches
The paper's claim is directly supported by a theorem in the formal canon.
supports
The theorem supports part of the paper's argument, but the paper may add assumptions or extra steps.
extends
The paper goes beyond the formal theorem; the theorem is a base layer rather than the whole result.
uses
The paper appears to rely on the theorem as machinery.
contradicts
The paper's claim conflicts with a theorem or certificate in the canon.
unclear
Pith found a possible connection, but the passage is too broad, indirect, or ambiguous to say the theorem truly supports the claim.

Reference graph

Works this paper leans on

22 extracted references · 22 canonical work pages · 2 internal anchors

  1. [2]

    emnlp-main.421/

    URL https://aclanthology.org/2023. emnlp-main.421/. Deas, N., Vente, B., Ananthram, A., Grieser, J. A., Patton, D. U., Kleiner, S., Iii, J. R. S., and McKeown, K. Data caricatures: On the representation of African American language in pretraining corpora. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T. (eds.),Proceed- ings of the 63rd Annual Mee...

    work page doi:10.18653/v1/2025.acl-long.1416 2023

  2. [4]

    Is llm-as- a-judge robust? investigating universal adversarial attacks on zero-shot llm assessment

    URL https://aclanthology.org/2025. acl-long.1252/. Fleisig, E., Smith, G., Bossi, M., Rustagi, I., Yin, X., and Klein, D. Linguistic bias in ChatGPT: Language models reinforce dialect discrimination. In Al-Onaizan, Y ., Bansal, M., and Chen, Y .-N. (eds.),Proceedings of the 2024 Conference on Empirical Methods in Nat- ural Language Processing, pp. 13541–1...

    work page doi:10.18653/v1/2024.emnlp-main 2025

  3. [5]

    Francesco D’Angelo, Maksym Andriushchenko, Aditya Vardhan Varre, and Nicolas Flammarion

    URL https://aclanthology.org/2024. emnlp-main.750/. Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y ., Ma, S., Liu, H., et al. A survey on llm-as-a- judge.The Innovation, 2024. Guo, D., Yang, D., Zhang, H., Song, J., Wang, P., Zhu, Q., Xu, R., Zhang, R., Ma, S., Bi, X., et al. Deepseek-r1 incentivizes reasoning in llms through reinf...

    work page doi:10.1038/s41586-025-09422-z 2024

  4. [6]

    findings-emnlp.913/

    URL https://aclanthology.org/2025. findings-emnlp.913/. Held, W., Ziems, C., and Yang, D. TADA : Task agnostic dialect adapters for English. In Rogers, A., Boyd-Graber, J., and Okazaki, N. (eds.),Findings of the Association for Computational Linguistics: ACL 2023, pp. 813–824, Toronto, Canada, July 2023. Association for Computa- tional Linguistics. doi: 1...

    work page doi:10.18653/v1/2023.findings-acl 2025

  5. [7]

    doi: 10.18653/v1/2024.findings-acl

    URL https://aclanthology.org/2023. findings-acl.51/. Hofmann, V ., Kalluri, P. R., Jurafsky, D., and King, S. Ai generates covertly racist decisions about people based on their dialect.Nature, 633(8028):147–154, 2024. Holt, F., Held, W., and Yang, D. Perceptions of lan- guage technology failures from South Asian English speakers. In Ku, L.-W., Martins, A....

    work page doi:10.18653/v1/2024.findings-acl 2023

  6. [8]

    findings-acl.241/

    URL https://aclanthology.org/2024. findings-acl.241/. Joshi, A., Dabre, R., Kanojia, D., Li, Z., Zhan, H., Haffari, G., and Dippold, D. Natural language processing for di- alects of a language: A survey.ACM Comput. Surv., 57(6), February 2025. ISSN 0360-0300. doi: 10.1145/3712060. URLhttps://doi.org/10.1145/3712060. Kortmann, B., Lunkenheimer, K., and Ehr...

    work page doi:10.1145/3712060 2024

  7. [9]

    Lin, F., Mao, S., La Malfa, E., Hofmann, V ., de Wynter, A., Wang, X., Chen, S.-Q., Wooldridge, M

    URL https://openreview.net/forum? id=YIpvHrQAks. Lin, F., Mao, S., La Malfa, E., Hofmann, V ., de Wynter, A., Wang, X., Chen, S.-Q., Wooldridge, M. J., Pierre- humbert, J. B., and Wei, F. Assessing dialect fairness and robustness of large language models in reasoning tasks. In Che, W., Nabende, J., Shutova, E., and Pilehvar, M. T. (eds.),Proceedings of th...

    work page

  8. [10]

    ISBN 979-8-89176-251-0

    Association for Computational Linguistics. ISBN 979-8-89176-251-0. doi: 10.18653/v1/2025.acl-long

    work page doi:10.18653/v1/2025.acl-long 2025

  9. [11]

    DeepSeek-V3 Technical Report

    URL https://aclanthology.org/2025. acl-long.317/. Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437, 2024. Liu, Y ., Held, W., and Yang, D. DADA: Dialect adap- tation via dynamic aggregation of linguistic rules. In Bouamor, H., Pino, J., and Ba...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/2023.emnlp-main 2025

  10. [12]

    B leu: a Method for Automatic Evaluation of Machine Translation

    URL https://aclanthology.org/2023. emnlp-main.850/. Mihalcea, R., Ignat, O., Bai, L., Borah, A., Chiruzzo, L., Jin, Z., Kwizera, C., Nwatu, J., Poria, S., and Solorio, T. Why ai is weird and shouldn’t be this way: To- wards ai for everyone, with everyone, by everyone.Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, 39(27):28657–28670, Ap...

    work page doi:10.3115/1073083.1073135 2023

  11. [13]

    Qwen3 Technical Report

    URL https://aclanthology.org/2023. emnlp-main.487/. Yang, A., Li, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Gao, C., Huang, C., Lv, C., et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388, 2025. Yong, Z.-X., Zhang, R., Forde, J. Z., Wang, S., Subramo- nian, A., Lovenia, H., Cahyawijaya, S., Winata, G. I., Sutawika, L., Cruz, J. C. B.,...

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.18653/v1/2022 2023

  12. [14]

    my object

    Feature is required in this locale.It would sound unnatural if the speaker used the original sentence instead of the transformed sentence. US example: Possessive: The transformed sentence uses to “my object” when the original sentence uses “me object” [Tips for Model Mirror? Column] For each feature listed, please evaluate LLM/voice agent’s capabilities i...

    work page

  13. [15]

    An LLM/voice agent should understand this term but not generate it

    Feature exists but is rare.The word/spelling might be understood but is outdated, region-specific, or influenced by other dialects. An LLM/voice agent should understand this term but not generate it. 3.Feature exists and is commonly used in colloquial speech.The word/spelling is widespread in informal Irish English but may coexist with other variants. Bot...

    work page

  14. [16]

    takeout” to “takeaway

    Feature is required in this locale.Using the US variant would sound unnatural or incorrect. The Irish variant is the standard and expected form. Examples: US IE note Rating Explanation (Optional) Excellent, Great Savage lexical Fruitcake Barmbrack lexical -or (e.g., color, favor) -our (e.g., colour, favour) orthographic Figure 6.Guidelines provided to Iri...

    work page 2023

  15. [17]

    V1 isgreatly preferredover V2

    work page

  16. [18]

    V1 issomewhat preferredover V2

    work page

  17. [19]

    No clear preference between V1 and V2

    work page

  18. [20]

    V2 issomewhat preferredover V1

    work page

  19. [21]

    y’all” instead of “you all

    V2 isgreatly preferredover V1 Order was randomly shuffled to ensure no annotation bias. We provide the following additional annotation instructions: For the USER annotations:Which version is more representative of how users from your country speak in day-to-day conversation? We are trying to capture colloquial expressions (e.g., “y’all” instead of “you al...

    work page 2024

  20. [22]

    The Conversation History (Conversation history of previous turns)

    work page

  21. [23]

    The Current User Turn (The message the Assistant must reply to)

    work page

  22. [24]

    morphosyntactic

    A list of Candidate Options for the Assistant’s response. Instructions: • Analyze the {sentence} of the ’Conversation History’ and ’Current User Turn’ to identify the specific English dialect being used. •Evaluate the ’Candidate Options’ and select the one that is linguistically consistent with the identified dialect from the conversation history and curr...

    work page