A novel constructed language with explicit grammar and dictionary reveals a large gap between human metalinguistic learning (87%) and the best LLM (47%) on translated CommonsenseQA.
Back to School: Translation Using Grammar Books
1 Pith paper cite this work. Polarity classification is still indexing.
1
Pith paper citing it
abstract
Machine translation systems for high resource languages perform exceptionally well and produce high quality translations. Unfortunately, the vast majority of languages are not considered high resource and lack the quantity of parallel sentences needed to train such systems. These under-represented languages are not without resources, however, and bilingual dictionaries and grammar books are available as linguistic reference material. With current large language models (LLMs) supporting near book-length contexts, we can begin to use the available material to ensure advancements are shared among all of the world's languages. In this paper, we demonstrate incorporating grammar books in the prompt of GPT-4 to improve machine translation and evaluate the performance on 16 topologically diverse low-resource languages, using a combination of reference material to show that the machine translation performance of LLMs can be improved using this method.
fields
cs.CL 1years
2025 1verdicts
CONDITIONAL 1representative citing papers
citing papers explorer
-
The Gold Medals in an Empty Room: Diagnosing Metalinguistic Reasoning in LLMs with Camlang
A novel constructed language with explicit grammar and dictionary reveals a large gap between human metalinguistic learning (87%) and the best LLM (47%) on translated CommonsenseQA.