The reviewed record of science sign in
Pith

arxiv: 2309.07423 · v1 · pith:FRFUXLBV · submitted 2023-09-14 · cs.CL

ChatGPT MT: Competitive for High- (but not Low-) Resource Languages

Nathaniel R. Robinson , Perez Ogayo , David R. Mortensen , Graham Neubig This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel pith:FRFUXLBVrecord.jsonopen to challenge →

classification cs.CL
keywords languageschatgptlanguagellmsanalysisevidenceexperimentallrls
0
0 comments X
read the original abstract

Large language models (LLMs) implicitly learn to perform a range of language tasks, including machine translation (MT). Previous studies explore aspects of LLMs' MT capabilities. However, there exist a wide variety of languages for which recent LLM MT performance has never before been evaluated. Without published experimental evidence on the matter, it is difficult for speakers of the world's diverse languages to know how and whether they can use LLMs for their languages. We present the first experimental evidence for an expansive set of 204 languages, along with MT cost analysis, using the FLORES-200 benchmark. Trends reveal that GPT models approach or exceed traditional MT model performance for some high-resource languages (HRLs) but consistently lag for low-resource languages (LRLs), under-performing traditional MT for 84.1% of languages we covered. Our analysis reveals that a language's resource level is the most important feature in determining ChatGPT's relative ability to translate it, and suggests that ChatGPT is especially disadvantaged for LRLs and African languages.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Nsanku: Evaluating Zero-Shot Translation Performance of LLMs for Ghanaian Languages

    cs.CL 2026-05 accept novelty 7.0

    Nsanku benchmark shows current LLMs achieve only modest zero-shot translation scores on 43 Ghanaian languages, with no model reaching both high average performance and high cross-language consistency.

  2. Finding Meaning in Embeddings: Concept Separation Curves

    cs.CL 2026-04 unverdicted novelty 6.0

    Concept Separation Curves provide a classifier-independent method to visualize and quantify how sentence embeddings distinguish conceptual meaning from syntactic variations across languages and domains.

  3. Evaluating Large Language Models for Hausa and Fongbe Machine Translation: Benchmarks, Failures, and Metric Reliability

    cs.CL 2026-06 unverdicted novelty 5.0

    LLM translation quality reaches acceptable human scores for Hausa but remains poor for Fongbe, automatic metrics show weak human correlation especially for Hausa due to neural embedding collapse, and at least 2,500 se...