REVIEW 7 cited by
Towards Multilingual LLM Evaluation for European Languages
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Towards Multilingual LLM Evaluation for European Languages
read the original abstract
The rise of Large Language Models (LLMs) has revolutionized natural language processing across numerous languages and tasks. However, evaluating LLM performance in a consistent and meaningful way across multiple European languages remains challenging, especially due to the scarcity of language-parallel multilingual benchmarks. We introduce a multilingual evaluation approach tailored for European languages. We employ translated versions of five widely-used benchmarks to assess the capabilities of 40 LLMs across 21 European languages. Our contributions include examining the effectiveness of translated benchmarks, assessing the impact of different translation services, and offering a multilingual evaluation framework for LLMs that includes newly created datasets: EU20-MMLU, EU20-HellaSwag, EU20-ARC, EU20-TruthfulQA, and EU20-GSM8K. The benchmarks and results are made publicly available to encourage further research in multilingual LLM evaluation.
Forward citations
Cited by 7 Pith papers
-
Meta-Learning Preferences for Multilingual LLM Alignment
Meta-learning a shared initialization on multilingual preference data lets LLMs align to a new language from ~100 preference samples, with up to 28% win-rate gains over baselines.
-
M\"OVE: A Holistic LLM Benchmark for the German Public Sector
MÖVE presents a new German-language benchmark evaluating 39 LLMs on performance and governance criteria using ten public-administration datasets.
-
KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Report
KletterMix is a translated German corpus from English pretraining data that yields measurable gains on German downstream tasks in controlled pretraining experiments.
-
CARTE: A Benchmark for Mapping Language Model Knowledge Across France
CARTE is a new benchmark for fine-grained regional knowledge in France that shows LLMs exhibit performance gaps across regions and scales, pointing to uneven pretraining coverage.
-
KletterMix: Climbing Toward High-Quality German Pretraining Data - The Full Report
A translated-English German corpus (725B tokens) produced higher point estimates on German HellaSwag and ARC-C than native German web corpora in matched 12B-token pretraining runs, though the differences are not stati...
-
From Data to Device: ELMOD An Efficient German-First 2.7B Language Model for Mobile Inference
A 2.7B German-first LLM trained cheaply on public data with language-specific quality filtering matches larger 7B models on German reasoning benchmarks and runs on-device.
-
Training LLMs on HPC Systems: Best Practices from the OpenGPT-X Project
Engineering report detailing HPC infrastructure, software choices, and performance measurements for training a 7B LLM using 3D parallelism on JUWELS Booster.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.