Pith. sign in

REVIEW 4 cited by

DIALECTBENCH: A NLP Benchmark for Dialects, Varieties, and Closely-Related Languages

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2403.11009 v2 pith:JO4LSOYF submitted 2024-03-16 cs.CL cs.AI

Fahim Faisal , Orevaoghene Ahia , Aarohi Srivastava , Kabir Ahuja , David Chiang , Yulia Tsvetkov , Antonios Anastasopoulos This is my paper · ORCID
classification cs.CLcs.AI
keywords languagevarietiesdialectbenchperformancebenchmarkcomprehensivedialectsevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Language technologies should be judged on their usefulness in real-world use cases. An often overlooked aspect in natural language processing (NLP) research and evaluation is language variation in the form of non-standard dialects or language varieties (hereafter, varieties). Most NLP benchmarks are limited to standard language varieties. To fill this gap, we propose DIALECTBENCH, the first-ever large-scale benchmark for NLP on varieties, which aggregates an extensive set of task-varied variety datasets (10 text-level tasks covering 281 varieties). This allows for a comprehensive evaluation of NLP system performance on different language varieties. We provide substantial evidence of performance disparities between standard and non-standard language varieties, and we also identify language clusters with large performance divergence across tasks. We believe DIALECTBENCH provides a comprehensive view of the current state of NLP for language varieties and one step towards advancing it further. Code/data: https://github.com/ffaisal93/DialectBench

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Using Contextually Aligned Online Reviews to Measure LLMs' Performance Disparities Across Language Varieties

    cs.CL 2025-02 conditional novelty 7.0 of 10

    LLMs predict sentiment worse on Taiwan Mandarin than Mainland Mandarin reviews, using a new contextually paired dataset from Booking.com.

  2. Lost in Variation? Evaluating NLI Performance in Basque and Spanish Geographical Variants

    cs.CL 2025-06 conditional novelty 6.0 of 10

    Language models are significantly worse at natural language inference when sentences are written in Basque or Spanish regional dialects, especially for Basque.

  3. AL-QASIDA: Analyzing LLM Quality and Accuracy Systematically in Dialectal Arabic

    cs.CL 2024-12 conditional novelty 6.0 of 10

    LLMs understand dialectal Arabic better than they generate it, and current post-training appears to bias them toward Modern Standard Arabic.

  4. Open or Closed LLM for Lesser-Resourced Languages? Lessons from Greek

    cs.CL 2025-01 conditional novelty 5.0 of 10

    Benchmarking Llama-70b against GPT-4o mini on Greek shows task-specific strengths, but the contamination-probe and legal-clustering claims need stronger baselines.

Pith tools