Vision Language Models (VLMs) often struggle with culture-specific knowledge, particularly in languages other than English and in underrepresented cultural contexts. To evaluate their understanding of such knowledge, we introduce WorldCuisines, a massive-scale benchmark for multilingual and multicultural, visually grounded language understanding. This benchmark includes a visual question answering (VQA) dataset with text-image pairs across 30 languages and dialects, spanning 9 language families and featuring over 1 million data points, making it the largest multicultural VQA benchmark to date. It includes tasks for identifying dish names and their origins. We provide evaluation datasets in two sizes (12k and 60k instances) alongside a training dataset (1 million instances). Our findings show that while VLMs perform better with correct location context, they struggle with adversarial contexts and predicting specific regional cuisines and languages. To support future research, we release a knowledge base with annotated food entries and images along with the VQA data.
WorldCuisines: A Massive-Scale Benchmark for Multilingual and Multicultural Visual Question Answering on Global Cuisines
WorldCuisines, a large-scale multilingual and multicultural visual question answering benchmark, evaluates Vision Language Models' understanding of cuisine-specific knowledge across various languages and cultural contexts.
- Year
- 2024
- Venue
- arXiv 2024
- Authors
- 51
- Hosting
- Abstract onlyARXIV-DEFAULT
Cite
Notes
Only stored in your browser.
Attribution
- Abstract & full text
- arxiv.org/abs/2410.12705v5ARXIV-DEFAULT
- TL;DR
- Semantic Scholar
Abstract
Authors
51Yi ZhouGenta Indra WinataRaj DabreTaro WatanabeAlham Fikri AjiPeerat LimkonchotiwatAlice OhShi-Xiong ZhangDavid Ifeoluwa AdelaniNedjma OusidhoumDavid AnugrahaHanyang ZhaoGarry KuwantoPatrick Amadeus IrawanJan Christian Blaise CruzEn-Shiun Annie LeeAyu PurwariantiDerry Tanti WijayaYutong WangSamuel CahyawijayaBryan WilieChong-Wah NgoHoly LoveniaFrederikus HudiRifki Afina PutriMuhammad Farid AdilazuardaHaryo Akbarianto WibowoLucky SusantoJunho MyungEnrico SantusAnar RzayevAdam NohejlUbaidillah Ariq PrathamaAfifa AmrianiAnirban DasAshmari PramodyaAulia AdilaCandy Olivia MawalimChing Lam ChengDaud AboladeEmmanuele ChersoniFariz IkhwantriJan Wira Gotama PutraMaria Angelica Riera MachinMarina ZhukovaMichael AnugrahaNatasha SantosaRio Alexander AudinoStephanie Yulia SalimYinxuan GuiShogo Okada