Why it matters

Encoder choice determines vision capability. Understanding shapes MLLM.

Advertisement

The architecture

ViT: transformer on image patches.

CLIP: image-text contrastive.

SigLIP: sigmoid loss variant.

Vision encoder stackPatches16x16 typicalViT / CLIPtransformer encoderEmbeddingsfor adapterSigLIP outperforms CLIP; EVA-CLIP scales further
Vision encoders.
Advertisement

How it works end to end

Resolution: 224/336/384/1024.

Patch size: 14/16.

Multi-resolution: AnyRes, dynamic.