We introduce MultiBLiMP 1.0, a massively multilingual benchmark of linguistic minimal pairs, covering 101 languages, 6 linguistic phenomena and containing more than 125,000 minimal pairs. Our minimal pairs are created using a fully automated pipeline, leveraging the large-scale linguistic resources of Universal Dependencies and UniMorph. MultiBLiMP 1.0 evaluates abilities of LLMs at an unprecedented multilingual scale, and highlights the shortcomings of the current state-of-the-art in modelling low-resource languages.
MultiBLiMP 1.0: A Massively Multilingual Benchmark of Linguistic Minimal Pairs
MultiBLiMP 1.0 evaluates large language models across 101 languages and 6 linguistic phenomena using a large dataset of minimal pairs, revealing challenges in modeling low-resource languages.
- Year
- 2025
- Venue
- arXiv 2025
- Authors
- 3
- Hosting
- Abstract onlyARXIV-DEFAULT
Cite
Notes
Only stored in your browser.
Attribution
- Abstract & full text
- arxiv.org/abs/2504.02768ARXIV-DEFAULT
- TL;DR
- Semantic Scholar