REVIEW 3 cited by
ChatGPT MT: Competitive for High- (but not Low-) Resource Languages
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Large language models (LLMs) implicitly learn to perform a range of language tasks, including machine translation (MT). Previous studies explore aspects of LLMs' MT capabilities. However, there exist a wide variety of languages for which recent LLM MT performance has never before been evaluated. Without published experimental evidence on the matter, it is difficult for speakers of the world's diverse languages to know how and whether they can use LLMs for their languages. We present the first experimental evidence for an expansive set of 204 languages, along with MT cost analysis, using the FLORES-200 benchmark. Trends reveal that GPT models approach or exceed traditional MT model performance for some high-resource languages (HRLs) but consistently lag for low-resource languages (LRLs), under-performing traditional MT for 84.1% of languages we covered. Our analysis reveals that a language's resource level is the most important feature in determining ChatGPT's relative ability to translate it, and suggests that ChatGPT is especially disadvantaged for LRLs and African languages.
Forward citations
Cited by 3 Pith papers
-
Dutch CrowS-Pairs: Adapting a Challenge Dataset for Measuring Social Biases in Language Models for Dutch
The paper presents a Dutch adaptation of the CrowS-Pairs bias benchmark and reports bias scores for seven masked and two autoregressive language models across nine demographic categories.
-
Toxicity-Aware Few-Shot Prompting for Low-Resource Singlish Translation
A two-stage pipeline combining human-curated Singlish examples with embedding-based LLM ranking selects GPT-4o mini for toxicity-preserving translation, reaching gold-level human scores for Chinese and Malay but not Tamil.
-
Bridging the Gap with Retrieval-Augmented Generation: Making Prosthetic Device User Manuals Available in Marginalised Languages
A proposed RAG-based framework for translating prosthetic device manuals into marginalised languages is described, but no results are presented.
Discussion (0). Sign in to comment.