One model, several data streams
A multimodal transformer typically maps every modality into the same embedding sequence — text becomes subword tokens, an image becomes a grid of patch embeddings, audio becomes frames — and a shared stack processes the concatenation. Because the parameters and the FLOP budget are shared, scaling is no longer a single curve L(N, D) but a vector of losses, one per modality, driven by how the fixed budget is split. The central questions become: does adding a second modality help the first (synergy) or hurt it (interference/competition)? And what data mixture minimizes the loss you actually care about?
Per-modality power laws
Each modality still follows a Chinchilla-style law in isolation — loss falls as a power of the tokens spent on it plus a modality-specific floor
L_m(D_m) = E_m + B_m / D_m^β_mbut the exponents β_m, floors E_m, and scales B_m differ by modality: images and text do not get cheaper at the same rate per token. The model shares N parameters across all of them, so a parameter spent representing images is not available for text. Multimodal scaling laws therefore fit a joint surface over (N, D_text, D_image, …), and the compute-optimal point balances the marginal loss reduction per FLOP across modalities.
Competition and synergy
Two forces pull against each other. Competition: modalities share capacity, so a fixed-size model split across many modalities is worse at each one than a specialist of the same size — the classic multi-task ‘taxation’. Synergy: grounding text in images (and vice versa) can improve representations, so a modality can be better with the other than alone, especially when its own data is limited. Which force wins depends on model scale and the mixture: small models are capacity-starved and feel competition; larger models have room for shared abstractions and are more likely to show synergy. This is the multimodal echo of the alignment/multi-task tax — a real cost that shrinks as capacity grows.
The mixture ratio is a hyperparameter
If a fraction α of training tokens are text and 1−α are images, then sweeping α traces a Pareto frontier: more text lowers text loss and raises image loss, and vice versa. There is no single ‘optimal’ mix — only the one that minimizes your weighted objective Σ_m w_m L_m, where w_m encodes how much you care about each modality. Because the per-modality exponents differ, the optimal α also shifts with scale: as the budget grows, the modality with the steeper power law keeps improving and deserves a changing share. Practical recipes often warm up on the abundant modality (text) and up-weight the scarcer one late in training.
A worked mixture example
Suppose at a fixed compute budget the fitted losses are L_text = 1.9 + 12/D_t^0.28 and L_img = 2.4 + 30/D_i^0.24 in nats, with a shared token budget D_t + D_i = 300B. An all-text run drives text loss down but leaves image loss at its floor-plus-large-term; a 50/50 split raises text loss modestly while cutting image loss sharply, because the image term 30/D_i^0.24 is large and responsive. Minimizing an equal-weight sum L_text + L_img lands at an interior mix (here skewed toward images, since their term dominates) — the exact point found by setting the marginal reductions equal: dL_text/dD_t = dL_img/dD_i under the budget constraint (a Lagrange condition identical in spirit to the compute-optimal derivation).
Image tokenization rewrites the token budget
The token count for images is not the number of images — it is the number of patches. A H×W image at patch size P yields (H/P)·(W/P) tokens: a 224×224 image at P=14 is 16×16 = 256 tokens; a 336×336 image is 576. Since attention is O(sequence^2), high-resolution vision inflates both the token budget and the per-example compute far faster than the pixel count suggests. This is why VLMs use tricks — smaller effective patch counts, token pooling/perceiver resamplers, or cropping — to keep image tokens from swamping text tokens. When you compute a modality’s D_m, count patches (and audio frames), not raw files.
Native vs bolted-on multimodality
Two architectures scale differently. Bolt-on designs freeze a pretrained vision encoder and a pretrained LLM and train a small connector (a projection or resampler) — cheap, and each tower keeps its own separately-scaled quality, but the fusion is shallow. Natively multimodal models train all modalities together from the start (or early), sharing the backbone — more expensive and more subject to competition, but capable of deeper cross-modal transfer and better joint reasoning at scale. The scaling story favors native training as budgets grow (synergy dominates), while bolt-on is the compute-efficient choice at smaller scale or when strong unimodal checkpoints already exist.
Audio, video, and the token explosion
Vision is not the worst offender — video is. A clip is a stack of frames, each a patch grid, so tokens scale as frames × patches_per_frame: even a few seconds at a modest frame rate can dwarf a long text document. Audio tokenized into frames (or discrete codec tokens) similarly produces long sequences for short clips. Because the transformer is quadratic in sequence length, these modalities force aggressive compression — temporal pooling, keyframe selection, learned tokenizers with small codebooks — or the compute budget is spent almost entirely on one modality. When you fit a data mixture, the per-modality D_m must be measured in these post-tokenization units, which is where naive file-count intuitions go badly wrong.