Multimodal LLM Training — Pipeline

Why it matters

MLLMs are frontier. Understanding pipeline shapes multimodal projects.

The architecture

Encoder: pretrained (CLIP, SigLIP, ViT).

Adapter: projects to LLM embedding space.

LLM: pretrained; may fine-tune.

How it works end to end

Stage 1: pretrain adapter (image-caption pairs).

Stage 2: instruction-tune on multimodal SFT.

Stage 3: optional alignment (DPO).