0

From Specific-MLLM to Omni-MLLM: A Survey about the MLLMs alligned with Multi-Modality

The transition from Specific-MLLM to Omni-MLLM enhances multimodal understanding and interaction by treating modalities as distinct languages, supported by pretraining and fine-tuning techniques.

Open
Year
2024
Venue
arXiv 2024
ArXiv
arxiv.org/abs/2412.11694
URL
arxiv.org/abs/2412.11694
Hosting
Abstract onlyARXIV-DEFAULT

Cite

Notes

Only stored in your browser.

Attribution

Abstract & full text
arxiv.org/abs/2412.11694ARXIV-DEFAULT
TL;DR
Semantic Scholar
Attribution policy →

Abstract

From the Specific-MLLM, which excels in single-modal tasks, to the Omni-MLLM, which extends the range of general modalities, this evolution aims to achieve understanding and generation of multimodal information. Omni-MLLM treats the features of different modalities as different "foreign languages," enabling cross-modal interaction and understanding within a unified space. To promote the advancement of related research, we have compiled 47 relevant papers to provide the community with a comprehensive introduction to Omni-MLLM. We first explain the four core components of Omni-MLLM for unified modeling and interaction of multiple modalities. Next, we introduce the effective integration achieved through "alignment pretraining" and "instruction fine-tuning," and discuss open-source datasets and testing of interaction capabilities. Finally, we summarize the main challenges facing current Omni-MLLM and outline future directions.