Why it matters

Multimodal serving is complex; tokenization + encoding dominate. Growing use case.

Advertisement

The architecture

Text tokenization.

Image encoding.

Cross-modal attention.

Multimodal servingText tokenstokenizerImage tokensvision encoderFusecross-modal attentionGPT-4V, Gemini, Claude 3, Llama 3.2 all multimodal; encoding costly
Multimodal serving.
Advertisement

How it works end to end

GPT-4V, Gemini, Claude 3.

Image encoder up front.

Fused attention.