Fundamentals

Multimodal

Multimodal models can handle different input types simultaneously. Modern models like Claude, GPT-4V, and Gemini accept both text and images. Some models also handle audio or video. Multimodality opens new applications: document analysis (OCR + reasoning), image understanding, content moderation, and more. However, multimodal models are more complex than text-only models. They require careful handling of different modalities, separate encoders for each type, and more computational resources. The quality of each modality varies—most multimodal models are strong on text and images, weaker on audio.