pith. machine review for the scientific record. sign in

arxiv: 2504.02768 · v4 · submitted 2025-04-03 · 💻 cs.CL

Recognition: unknown

MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs

Jaap Jumelet , Leonie Weissweiler , Joakim Nivre , Arianna Bisazza
Authors on Pith no claims yet
classification 💻 cs.CL
keywords minimalpairslinguisticmultiblimpmultilingualbenchmarklanguagesmassively
0
0 comments X
read the original abstract

We introduce MultiBLiMP 1.0, a massively multilingual benchmark of linguistic minimal pairs, covering 101 languages and 2 types of subject-verb agreement, containing more than 128,000 minimal pairs. Our minimal pairs are created using a fully automated pipeline, leveraging the large-scale linguistic resources of Universal Dependencies and UniMorph. MultiBLiMP 1.0 evaluates abilities of LLMs at an unprecedented multilingual scale, and highlights the shortcomings of the current state-of-the-art in modelling low-resource languages.

This paper has not been read by Pith yet.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Implicit Representations of Grammaticality in Language Models

    cs.CL 2026-05 unverdicted novelty 7.0

    Linear probes on LM hidden states detect grammaticality better than string probabilities, generalize to human benchmarks and other languages, and correlate weakly with likelihood.