Why it matters

Video LLMs are frontier + hard. Understanding shapes approach.

Advertisement

The architecture

Frame sampling: uniform or content-aware.

Encoder: per-frame or video-native.

Temporal tokens or aggregation.

Video LLM structureFrame samplingN frames per videoPer-frame encoderor video-nativeTemporal fusionattention or poolFrame sampling + memory limits are core constraints
Video LLM.
Advertisement

How it works end to end

Frame sampling: uniform or importance.

Per-frame: ViT + concat + LLM.

Video-native: VideoMAE + temporal fusion.

Long video: memory tokens.