Multimodal Scaling

One model, several data streams

A multimodal transformer typically maps every modality into the same embedding sequence — text becomes subword tokens, an image becomes a grid of patch embeddings, audio becomes frames — and a shared stack processes the concatenation. Because the parameters and the FLOP budget are shared, scaling is no longer a single curve L(N, D) but a vector of losses, one per modality, driven by how the fixed budget is split. The central questions become: does adding a second modality help the first (synergy) or hurt it (interference/competition)? And what data mixture minimizes the loss you actually care about?

Per-modality power laws

Each modality still follows a Chinchilla-style law in isolation — loss falls as a power of the tokens spent on it plus a modality-specific floor

L_m(D_m) = E_m + B_m / D_m^β_m

but the exponents β_m, floors E_m, and scales B_m differ by modality: images and text do not get cheaper at the same rate per token. The model shares N parameters across all of them, so a parameter spent representing images is not available for text. Multimodal scaling laws therefore fit a joint surface over (N, D_text, D_image, …), and the compute-optimal point balances the marginal loss reduction per FLOP across modalities.

Competition and synergy

Two forces pull against each other. Competition: modalities share capacity, so a fixed-size model split across many modalities is worse at each one than a specialist of the same size — the classic multi-task ‘taxation’. Synergy: grounding text in images (and vice versa) can improve representations, so a modality can be better with the other than alone, especially when its own data is limited. Which force wins depends on model scale and the mixture: small models are capacity-starved and feel competition; larger models have room for shared abstractions and are more likely to show synergy. This is the multimodal echo of the alignment/multi-task tax — a real cost that shrinks as capacity grows.

The mixture ratio is a hyperparameter

If a fraction α of training tokens are text and 1−α are images, then sweeping α traces a Pareto frontier: more text lowers text loss and raises image loss, and vice versa. There is no single ‘optimal’ mix — only the one that minimizes your weighted objective Σ_m w_m L_m, where w_m encodes how much you care about each modality. Because the per-modality exponents differ, the optimal α also shifts with scale: as the budget grows, the modality with the steeper power law keeps improving and deserves a changing share. Practical recipes often warm up on the abundant modality (text) and up-weight the scarcer one late in training.

A worked mixture example

Suppose at a fixed compute budget the fitted losses are L_text = 1.9 + 12/D_t^0.28 and L_img = 2.4 + 30/D_i^0.24 in nats, with a shared token budget D_t + D_i = 300B. An all-text run drives text loss down but leaves image loss at its floor-plus-large-term; a 50/50 split raises text loss modestly while cutting image loss sharply, because the image term 30/D_i^0.24 is large and responsive. Minimizing an equal-weight sum L_text + L_img lands at an interior mix (here skewed toward images, since their term dominates) — the exact point found by setting the marginal reductions equal: dL_text/dD_t = dL_img/dD_i under the budget constraint (a Lagrange condition identical in spirit to the compute-optimal derivation).

Image tokenization rewrites the token budget

The token count for images is not the number of images — it is the number of patches. A H×W image at patch size P yields (H/P)·(W/P) tokens: a 224×224 image at P=14 is 16×16 = 256 tokens; a 336×336 image is 576. Since attention is O(sequence^2), high-resolution vision inflates both the token budget and the per-example compute far faster than the pixel count suggests. This is why VLMs use tricks — smaller effective patch counts, token pooling/perceiver resamplers, or cropping — to keep image tokens from swamping text tokens. When you compute a modality’s D_m, count patches (and audio frames), not raw files.

Native vs bolted-on multimodality

Two architectures scale differently. Bolt-on designs freeze a pretrained vision encoder and a pretrained LLM and train a small connector (a projection or resampler) — cheap, and each tower keeps its own separately-scaled quality, but the fusion is shallow. Natively multimodal models train all modalities together from the start (or early), sharing the backbone — more expensive and more subject to competition, but capable of deeper cross-modal transfer and better joint reasoning at scale. The scaling story favors native training as budgets grow (synergy dominates), while bolt-on is the compute-efficient choice at smaller scale or when strong unimodal checkpoints already exist.

Audio, video, and the token explosion

Vision is not the worst offender — video is. A clip is a stack of frames, each a patch grid, so tokens scale as frames × patches_per_frame: even a few seconds at a modest frame rate can dwarf a long text document. Audio tokenized into frames (or discrete codec tokens) similarly produces long sequences for short clips. Because the transformer is quadratic in sequence length, these modalities force aggressive compression — temporal pooling, keyframe selection, learned tokenizers with small codebooks — or the compute budget is spent almost entirely on one modality. When you fit a data mixture, the per-modality D_m must be measured in these post-tokenization units, which is where naive file-count intuitions go badly wrong.

Interleaved vs paired data

The structure of multimodal data also shapes scaling. Paired data (an image with its caption) teaches tight alignment and is what contrastive and captioning objectives need. Interleaved data (documents where images and text alternate naturally, like web pages) teaches in-context, document-level reasoning across modalities and tends to unlock few-shot multimodal behavior. The two are not interchangeable: a model trained only on tight pairs can caption but struggles to reason over a mixed document, while interleaved data alone gives weaker fine-grained grounding. In practice the mixture spans both axes — which modalities, and in which structure — and each interacts with scale. A further wrinkle is that paired data is comparatively scarce and expensive to curate, while interleaved data is abundant on the web, so the effective D_m you can reach differs by data type as well as by modality — another quantity the mixture search must respect.

Cross-modal transfer

A striking multimodal scaling effect is transfer: capability learned in one modality partly carries to another. Text reasoning can improve a model’s image question-answering, and paired data teaches alignment that neither modality could learn alone. In effective-data terms (see the transfer-scaling sibling), one modality’s tokens are worth some effective tokens of the other — a bonus that partly offsets the competition tax. This is why scarce modalities (e.g. specialized medical images) benefit most from joint training with an abundant one: they inherit structure they could never learn from their own thin data.

Pitfalls and the practical takeaway

Common traps: measuring image ‘data’ in files rather than patches (undercounting compute); using a single learning rate/mix tuned at small scale and assuming it transfers (the optimal mix moves with budget); and reading a modality’s isolated loss as if the shared model behaved like a specialist (it does not — account for competition); and comparing two multimodal models on a single modality’s benchmark while ignoring that they used different mixtures, which confounds architecture with data-allocation choices. The operational recipe is: fit each modality’s power law, express your goal as a weighted loss, then choose the mixture and resolution that equalize marginal loss-per-FLOP across modalities at your scale — and revisit that mix as you scale up, because the balance point moves. Treat resolution, patch size, and the paired-vs-interleaved split as part of that same search: each one silently changes how many tokens a modality actually consumes, and therefore where the compute-optimal balance falls.