Pith. sign in

REVIEW 4 major objections 5 minor 33 references

Using Contextually Aligned Online Reviews to Measure LLMs' Performance Disparities Across Language Varieties

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read LLMs consistently underperform on Taiwan Mandarin reviews even when hotel, rating, and length are matched.

desk verdict A genuinely new method for cross-variety benchmarking, but the headline claim of consistent TW underperformance is weakened by the authors' own human study showing TW reviews are harder for humans too. read the letter →

arxiv 2502.07058 v3 pith:QQ3PWKE7 submitted 2025-02-10 cs.CL cs.HC

Zixin Tang , Chieh-Yang Huang , Tsung-Che Li , Ho Yin Sam Ng , Hen-Hsen Huang , Ting-Hao 'Kenneth' Huang This is my paper · ORCID
classification cs.CLcs.HC
keywords languagevarietiesTaiwanMandarinMainlandsentimentanalysislargemodelsdialectbiascontextualalignmenthotelreviews
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that international hotel-review platforms like Booking.com can be mined to compare how well large language models handle different varieties of the same language, without the usual cost of hiring native speakers to construct parallel examples. It pairs reviews written in Taiwan Mandarin with reviews written in Mainland Mandarin for the same hotel, the same coarse rating class, and a similar text length, then asks six LLMs to predict each review's 1-10 rating. Across 54 model-and-format experiments, models were significantly worse on Taiwan Mandarin reviews in the large majority of cases; the gap grew when the review's section structure was removed and was largest on short texts. The paper's larger claim is that this low-cost, contextually aligned design can be reused across many language varieties to expose where LLMs still fail.

What carries the argument

The load-bearing mechanism is the contextually aligned review pair: each pair contains one Taiwan Mandarin and one Mainland Mandarin review from the same hotel, falling in the same coarse rating class (1-3, 4-7, 8-10) and in the same 10-token text-length bin. This pairing is meant to give instance-level comparability between language varieties without the expensive native-speaker translation or annotation used in earlier efforts. The experimental apparatus then varies input format - structured (title/positive/negative), plain (concatenated), or shuffled (random order) - to see how much of the variety gap depends on format cues, and compares GPT-4o, Llama3 (8B, 70B, 405B), and Gemma2 (9B, 27B).

What would settle it

A decisive check would be a re-run in which every Taiwan Mandarin review is paired with a Mainland Mandarin review produced by professional human translation of the same content, holding hotel, rating class, length, and format constant; if the performance gap disappears on such semantically identical pairs, the reported disparity is attributable to content rather than to language variety, and the paper's central claim fails.

Watch

Extended reading notes

Core claim

The central discovery is that LLMs consistently underperform on Taiwan Mandarin compared with Mainland Mandarin on a rating-prediction sentiment task built from naturally occurring hotel reviews. In the paper's results, 38 of 54 experiments showed significant accuracy differences between the varieties and 47 of 54 showed significant differences in mean squared error; among those significant differences, 36 of 38 accuracy gaps and 45 of 47 MSE gaps indicated worse performance on Taiwan Mandarin. The authors also report that the gap widens when inputs lose their section structure (plain or shuffled versus structured) and that shorter reviews show larger MSE gaps, while machine translation from Taiwan to Mainland Mandarin improves predictions but the reverse direction does not. A human-participant study found that native speakers also predicted ratings less accurately for Taiwan Mandarin reviews, which the paper takes as a caution that part of the gap may reflect task difficulty or content rather than a language-variety-specific deficit.

Load-bearing premise

The argument collapses if matched reviews still differ in something other than language variety, such as how explicitly the writer states their opinion, what they complain about, or how they use star ratings, because then the performance gap could come from task difficulty rather than from the model's handling of the variety itself.

Editorial extensions

If this is right

  • Six leading LLMs are currently less reliable at predicting Taiwan Mandarin reviews than Mainland Mandarin ones on matched hotel-review pairs.
  • The reliability gap grows when the model sees a plain paragraph or shuffled sections rather than the original title/positive/negative structure.
  • Short Taiwan Mandarin reviews are where the model's mean squared error is largest relative to Mainland Mandarin, so low-context user text is the riskiest input.
  • Machine translation between the two varieties is not a neutral pairing tool: TW-to-CN translation improves predictions, while CN-to-TW does not.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same 'same hotel, same rating class, same length' pairing could be exported to other varieties with large review corpora, such as Brazilian versus European Portuguese or British versus American English, creating near-free dialect audits for future models.
  • Since native-speaker raters also scored Taiwan Mandarin lower, the study implies that any benchmark that reports an LLM gap should include a human-difficulty control; otherwise the gap cannot be attributed to the model.
  • A testable follow-up is to calibrate each model's outputs on a per-variety rating distribution: if the Taiwan gap shrinks after calibration, part of the disparity is a scoring-scale artifact rather than a language-understanding failure.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes using contextually aligned online hotel reviews to benchmark LLM performance across language varieties. The authors collect 22,918 review pairs from Booking.com, each containing one Taiwan Mandarin (TW) and one Mainland Mandarin (CN) review of the same hotel with ratings in the same coarse 3-class bin and similar text length. Six LLMs (GPT-4o, three Llama3 sizes, two Gemma2 sizes) predict the 1-10 rating under structured, plain, and shuffled input formats. The authors report that in most model/setting/length combinations, accuracy is lower and MSE higher for TW reviews, that disrupting review structure widens the gap, that shorter reviews show larger MSE gaps, that machine translation from TW to CN improves GPT-4o performance while the reverse direction is mixed, and that human raters also predict CN reviews more accurately than TW reviews. Confound analyses address writing quality, code-mixing, and score distributions. The central claim stated in the abstract is that 'LLMs consistently underperform in Taiwan Mandarin.'

Significance. The dataset construction method is the paper's core contribution, and it is a real one: pairing reviews by the same hotel, coarse rating class, and length bin yields 22,918 TW-CN pairs at a fraction of the cost of native-speaker instance-level translation, and the code and data are released. The paper is notably transparent about confounds: it reports human validation of writing quality and content-rating agreement, code-mixing rates, score distributions, an MT directionality experiment, and a human rating-prediction study. The aggregate empirical pattern - 36 of 38 significant accuracy gaps and 45 of 47 significant MSE gaps favoring Mainland Mandarin across six models and three input formats - is a substantial and falsifiable finding, and if it survives the confound-control analyses suggested below it would justify the abstract's claim.

major comments (4)
  1. [Section 5; Appendix B.2] The authors' own human study shows that native speakers also predict ratings significantly better for Mainland Mandarin reviews than for Taiwan Mandarin reviews (accuracy 37.00% vs. 28.75%, p=.016; MSE 2.795 vs. 3.510, p=.036), and the paper labels this confound 'Plausible.' Yet the abstract and Section 7 nonetheless assert that 'LLMs consistently underperform in Taiwan Mandarin.' As it stands, the evidence establishes a performance disparity on a contextually aligned rating-prediction task, not an LLM-specific bias: the TW reviews in the paired sample appear intrinsically harder to predict, for humans and models alike. Because this attribution is the paper's headline claim, the point is load-bearing. The paper should either (i) reframe the claim as a task-level disparity, or (ii) provide a direct comparison of model gaps with human gaps on identical pairs, or an analysis stratifying pairs by human-rated predictability, to support the bias reading. The post-hoc exclusion of the two lowest-performing participants (both Taiwan-Mandarin speakers, Appendix B.2) further complicates the human baseline and should be justified or analyzed with and without exclusion.
  2. [Table 1, rows for Gemma2] The word 'consistently' is contradicted by the paper's own data. In the structured-short condition, Gemma2 9B and Gemma2 27B show significantly better accuracy on TW than on CN (Delta Acc = -1.24** and -1.07**), and in 8 of the 54 model/setting/length cells in Table 1 accuracy is numerically higher for TW (e.g., Llama3-8B structured overall, Delta Acc = -1.01). Of the 38 statistically significant accuracy gaps, 2 favor TW. The direction of the disparity therefore depends on model, input format, and metric; 'consistently' overstates the pattern. The finding should be reported as a majority pattern with the counts (e.g., 36 of 38 significant accuracy comparisons and 45 of 47 significant MSE comparisons favor CN), and the abstract's wording should be aligned with that.
  3. [Appendix D; Section 4] Invalid predictions are excluded rather than modeled, and the invalid counts reported in Table 6 are substantial and model-dependent: Llama3-8B has 168 invalid predictions in the plain condition and 139 in the structured condition, and Llama3-405B has 130 invalid in the structured condition. Because the analysis keeps only pairs in which both members produced a valid score, any systematic relationship between invalidity and language variety would bias the estimated gap; for example, if TW reviews more often trigger off-range or non-numeric outputs, dropping those pairs would differentially remove hard TW items. The paper should report invalid counts separately by variety and setting, and include a sensitivity analysis (e.g., treating invalid outputs as a separate outcome class or imputing them) to demonstrate that the headline gaps are robust to the exclusion mechanism.
  4. [Section 4; Tables 1-2] The 54 experiments are reported with per-cell significance stars and no correction for multiple comparisons; at alpha=.05 one expects roughly 2.7 spurious significant cells by chance. The aggregate directional counts (36/38 and 45/47) are clearly robust to this concern, but load-bearing secondary claims are not. In particular, the claim that 'shorter reviews tend to produce larger MSE gaps' is inferred from comparing the pattern of stars across rows of Table 2 rather than from a test of the length by variety interaction; the paper should confirm this with a model that includes the interaction (with model as a random effect, or per-model interaction tests), and should also state which statistical test produced the reported p-values (paired t-test, McNemar, or bootstrap).
minor comments (5)
  1. [Table 2 caption] The caption says the significance notation and color coding 'follow the same conventions as in Table 2'; this should refer to Table 1.
  2. [Section 1] The opening sentence, 'Of the world's 7,000 languages, sixty (60) million people speak British English, 23 million speak Taiwan Mandarin, and 10 million speak European Portuguese, compared to...' is ungrammatical and appears to be a damaged list item; please rewrite it.
  3. [Section 3.2] It is not stated whether the Mainland Mandarin reviews were validated by native Mainland Mandarin speakers or by the same five Taiwan Mandarin raters; this matters for interpreting the small quality differences, so rater backgrounds should be reported for both groups.
  4. [Appendix B.1] Participants were shown 'both the English text and its translation'; since the study evaluates Chinese reviews, please clarify why the English text was displayed and whether it could influence the quality and agreement judgments.
  5. [Appendix G] The text refers to 'presented in 9' where Table 9 is meant.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark claim is an empirical measurement against externally assigned hotel ratings, with no fitted parameter renamed as a prediction and no load-bearing self-citation.

full rationale

The paper's central claim ('LLMs consistently underperform in Taiwan Mandarin') is an empirical result derived from a task with independently assigned ground truth: Booking.com guests assigned the 1-10 ratings that the LLMs are asked to predict. No quantity used in the analysis is defined in terms of the outcome, and no fitted parameter is renamed as a prediction. The pairing criteria in Section 3.1 (same hotel, 3-class rating bin, 10-token length bin) are data-curation choices made before model evaluation; the paper's own pilot study (Appendix E) is used only to decide how to stratify results by text length, not to tune or derive the reported accuracy/MSE gaps. The human study in Section 5 does show that native speakers also perform worse on Taiwan Mandarin reviews, which is a legitimate threat to the interpretation that the gap is LLM-specific, but that is a correctness/confounding concern, not a circularity concern. There are no self-citations used as load-bearing evidence for the central claim, and the prior work cited (e.g., Ziems et al. for dialect disparities in English) is external and does not presuppose the paper's conclusion. Consequently, the derivation is self-contained with respect to the measured phenomenon, and no circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's analysis rests on three hand-chosen pairing thresholds (rating classes, length bins, short/long cut) and four domain assumptions about label reliability, contextual alignment, task validity, and exclusion validity. No new entities are introduced.

free parameters (3)
  • Rating class bins = 1-3 negative, 4-7 neutral, 8-10 positive
    Hand-chosen in Section 3.1 to pair reviews with 'similar' sentiment; coarse classes allow many pairs but tolerate rating differences within a class.
  • Length bin size and cap = 10-token bins, 500-token cap
    Hand-chosen in Section 3.1 to pair reviews of similar length; the cap excludes long reviews and may affect the distribution.
  • Short/long threshold = 49 vs 50 Chinese characters
    Chosen after a pilot study (Appendix E) to separate text lengths; the threshold is post hoc.
assumptions (4)
  • domain assumption User self-reported nationality/region corresponds to language variety
    Section 3.1 uses 'nationality/region' labels to assign TW vs CN; Section 6 acknowledges this is an imperfect proxy.
  • domain assumption Paired reviews (same hotel, rating class, length bin) are contextually aligned enough to attribute score-prediction gaps to variety
    Section 3.1 defines pairing criteria; Section 8 lists residual confounds as a limitation.
  • domain assumption Rating prediction on these reviews is a valid probe of language-variety performance
    Section 4 sets the task; the authors note in Section 5 that predicting scores from content is not a typical human task.
  • domain assumption Excluding invalid model outputs does not introduce bias
    Appendix D excludes pairs with invalid predictions; if invalidity correlates with variety, the comparison is biased.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Using Contextually Aligned Online Reviews to Measure LLMs' Performance Disparities Across Language Varieties." pith.science (2026). https://pith.science/paper/QQ3PWKE7

@misc{pith2026250207058,
  author       = {Pith},
  title        = {Pith review of: Using Contextually Aligned Online Reviews to Measure LLMs' Performance Disparities Across Language Varieties},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QQ3PWKE7}},
  note         = {Machine review of arXiv:2502.07058}
}
read the original abstract

A language can have different varieties. These varieties can affect the performance of natural language processing (NLP) models, including large language models (LLMs), which are often trained on data from widely spoken varieties. This paper introduces a novel and cost-effective approach to benchmark model performance across language varieties. We argue that international online review platforms, such as Booking.com, can serve as effective data sources for constructing datasets that capture comments in different language varieties from similar real-world scenarios, like reviews for the same hotel with the same rating using the same language (e.g., Mandarin Chinese) but different language varieties (e.g., Taiwan Mandarin, Mainland Mandarin). To prove this concept, we constructed a contextually aligned dataset comprising reviews in Taiwan Mandarin and Mainland Mandarin and tested six LLMs in a sentiment analysis task. Our results show that LLMs consistently underperform in Taiwan Mandarin.

Figures

Figures reproduced from arXiv: 2502.07058 by the authors.

Figure 1
Figure 1. Online review platforms can be data sources [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Impact of text length on sentiment classification performance. The top graph shows accuracy, and the [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Comparison of accuracy between Mainland Mandarin and Taiwan Mandarin for short (left) and long [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Comparison of MSE between Mainland Mandarin and Taiwan Mandarin for short (left) and long (right) [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 23 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Nava, Matteo Calabrese, Jean-Marc Christille, and Chiara B

    Marco Alderighi, Consuelo R. Nava, Matteo Calabrese, Jean-Marc Christille, and Chiara B. Salvemini. 2022. https://doi.org/10.1016/j.jbusres.2022.03.017 Consumer perception of price fairness and dynamic pricing: Evidence from booking.com . Journal of Business Research, 145:769--783

    work page doi:10.1016/j.jbusres.2022.03.017 2022

  4. [4]

    Jeremy Barnes, Toni Badia, and Patrik Lambert. 2018. https://aclanthology.org/L18-1104 M ulti B ooked: A corpus of B asque and C atalan hotel reviews annotated for aspect-level sentiment classification . In Proceedings of the Eleventh International Conference on Language Resources and Evaluation ( LREC 2018) , Miyazaki, Japan. European Language Resources ...

    work page 2018

  5. [5]

    Su Lin Blodgett, Solon Barocas, Hal Daum \'e III, and Hanna Wallach. 2020. Language (technology) is power: A critical survey of “bias” in nlp. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5454--5476

    work page 2020

  6. [6]

    Su Lin Blodgett and Brendan O'Connor. 2017. Racial disparity in natural language processing: A case study of social media african-american english. arXiv preprint arXiv:1707.00061

    arXiv 2017

  7. [7]

    emoji : emoji terminal output for python

    carpedm20. emoji : emoji terminal output for python. https://github.com/carpedm20/emoji

    work page

  8. [8]

    Po-Heng Chen, Sijia Cheng, Wei-Lin Chen, Yen-Ting Lin, and Yun-Nung Chen. 2024. https://openreview.net/forum?id=7jSMMvXLri Measuring taiwanese mandarin language understanding . In First Conference on Language Modeling

    work page 2024

Show all 33 references
  1. [9]

    Mucong Ding, Chenghao Deng, Jocelyn Choo, Zichu Wu, Aakriti Agrawal, Avi Schwarzschild, Tianyi Zhou, Tom Goldstein, John Langford, Anima Anandkumar, and Furong Huang. 2024. https://openreview.net/forum?id=iNB4uoFQJb Easy2hard-bench: Standardized difficulty labels for profiling...

    2024

  2. [10]

    Fahim Faisal, Orevaoghene Ahia, Aarohi Srivastava, Kabir Ahuja, David Chiang, Yulia Tsvetkov, and Antonios Anastasopoulos. 2024. Dialectbench: A nlp benchmark for dialects, varieties, and closely-related languages. arXiv preprint arXiv:2403.11009

    2024 arXiv

  3. [11]

    Shahriar Golchin, Nikhil Garuda, Christopher Impey, and Matthew Wenger. 2025. https://aclanthology.org/2025.coling-main.263/ Grading massive open online courses using large language models . In Proceedings of the 31st International Conference on Computational Linguistics, page...

    2025

  4. [12]

    Sophie Groenwold, Lily Ou, Aesha Parekh, Samhita Honnavalli, Sharon Levy, Diba Mirza, and William Yang Wang. 2020. Investigating african-american vernacular english in transformer-based text generation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lang...

    2020

  5. [13]

    Dirk Hovy and Anders Johannsen. 2016. https://aclanthology.org/L16-1477 Exploring language variation across E urope - a web-based tool for computational sociolinguistics . In Proceedings of the Tenth International Conference on Language Resources and Evaluation ( LREC '16) , p...

    2016

  6. [14]

    Aditya Joshi, Raj Dabre, Diptesh Kanojia, Zhuang Li, Haolan Zhan, Gholamreza Haffari, and Doris Dippold. 2024. Natural language processing for dialects of a language: A survey. arXiv preprint arXiv:2401.05632

    2024 arXiv

  7. [15]

    Anjali Kantharuban, Ivan Vuli \'c , and Anna Korhonen. 2023. Quantifying the dialect gap and its correlates across languages. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 7226--7245

    2023

  8. [16]

    Masamune Kobayashi, Masato Mita, and Mamoru Komachi. 2024. https://aclanthology.org/2024.bea-1.6/ Large language models are state-of-the-art evaluator for grammatical error correction . In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Appli...

    2024

  9. [17]

    Sachin Kumar, Antonios Anastasopoulos, Shuly Wintner, and Yulia Tsvetkov. 2021. https://doi.org/10.18653/v1/2021.acl-short.16 Machine translation into low-resource language varieties . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics a...

    2021 doi

  10. [18]

    Taja Kuzman, Peter Rupnik, and Nikola Ljube s i \'c . 2023. Get to know your parallel data: Performing english variety and genre classification over macocu corpora. In Tenth Workshop on NLP for Similar Languages, Varieties and Dialects (VarDial 2023), pages 91--103

    2023

  11. [19]

    Alexander Kwako, Yixin Wan, Jieyu Zhao, Mark Hansen, Kai-Wei Chang, and Li Cai. 2023. Does bert exacerbate gender or l1 biases in automated english speaking assessment? In Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023...

    2023

  12. [20]

    Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu, and James Zou. 2023. Gpt detectors are biased against non-native english writers. Patterns, 4(7)

    2023

  13. [21]

    Brandon Lwowski, Paul Rad, and Anthony Rios. 2022. Measuring geographic performance disparities of offensive language classifiers. In Proceedings of the 29th International Conference on Computational Linguistics, pages 6600--6616

    2022

  14. [22]

    Millicent Ochieng, Varun Gumma, Sunayana Sitaram, Jindong Wang, Keshet Ronen, Kalika Bali, and Jacki O'Neill. 2024. Beyond metrics: Evaluating llms' effectiveness in culturally nuanced, low-resource real-world scenarios. arXiv preprint arXiv:2406.00343

    2024 arXiv

  15. [23]

    Taku Sakamoto, Saku Sugawara, and Akiko Aizawa. 2025. https://aclanthology.org/2025.coling-main.666/ Development of numerical error detection tasks to analyze the numerical capabilities of language models . In Proceedings of the 31st International Conference on Computational L...

    2025

  16. [24]

    Yohei Seki, David Kirk Evans, Lun-Wei Ku, Hsin-Hsi Chen, Noriko Kando, Chin-Yew Lin, et al. 2007. Overview of opinion analysis pilot task at ntcir-6. In NTCIR

    2007

  17. [25]

    Rickard Stureborg, Dimitris Alikaniotis, and Yoshi Suhara. 2024. Large language models are inconsistent and biased evaluators. arXiv preprint arXiv:2405.01724

    2024 arXiv

  18. [26]

    Tmmlu+: An improved traditional chinese evaluation suite for foundation models

    Zhi Rui Tam, Ya Ting Pai, Yen-Wei Lee, Hong-Han Shuai, Jun-Da Chen, Wei Min Chu, and Sega Cheng. Tmmlu+: An improved traditional chinese evaluation suite for foundation models. In First Conference on Language Modeling

  19. [27]

    Zhon: Constants used in chinese text processing

    tsroten. Zhon: Constants used in chinese text processing. https://github.com/tsroten/zhon

  20. [28]

    Unicode character database

    Unicode. Unicode character database. https://www.unicode.org/reports/tr44/

  21. [29]

    Marcos Zampieri, Shervin Malmasi, Yves Scherrer, Tanja Samardzic, Francis Tyers, Miikka Silfverberg, Natalia Klyueva, Tung-Le Pan, Chu-Ren Huang, Radu Tudor Ionescu, et al. 2019. A report on the third vardial evaluation campaign. In Proceedings of the Sixth Workshop on NLP for...

    2019

  22. [30]

    Marcos Zampieri, Preslav Nakov, and Yves Scherrer. 2020. https://doi.org/10.1017/S1351324920000492 Natural language processing for similar languages, varieties, and dialects: A survey . Natural Language Engineering, 26(6):595–612

    2020 doi

  23. [31]

    Ruochen Zhang, Samuel Cahyawijaya, Jan Christian Blaise Cruz, Genta Winata, and Alham Fikri Aji. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.774 Multilingual large language models are not (yet) code-switchers . In Proceedings of the 2023 Conference on Empirical Methods i...

    2023 doi

  24. [32]

    Caleb Ziems, Jiaao Chen, Camille Harris, Jessica Anderson, and Diyi Yang. 2022. Value: Understanding dialect disparity in nlu. arXiv preprint arXiv:2204.03031

    2022 arXiv

  25. [33]

    Caleb Ziems, William Held, Jingfeng Yang, Jwala Dhamala, Rahul Gupta, and Diyi Yang. 2023. Multi-value: A framework for cross-dialectal english nlp. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 744--768

    2023

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.