MULTIMODAL INPUT · RESOLUTION AND COST
Gemini multimodal input: send pixels when the question needs them
Ask three questions about a chart and decide whether to send OCR text, pixels or both, at what size and crop, and what that costs in tokens.
Your experiment
The source is a 2400 x 1600 dashboard screenshot whose Q3 value is printed in 9-pixel text. The opening request sends the image resized to 768 pixels. Find the cheapest input that keeps each question answerable.
Play replays the computed run over sixty inspection seconds; Next event steps, Restart rewinds, and Inspection time scrubs. Any input change recomputes the run and rewinds it without autoplay.
Event trace
Computed data
Final metrics
Fixture assumptions: image tokens follow the rule documented for the Gemini models described in the linked article (258 tokens if both sides are at most 384 px, otherwise 258 per 768 x 768 tile); newer models and media-resolution settings count differently. Text is treated as readable at 6 px glyph height and bar heights as distinguishable at 3 px. The OCR transcript has the caption and axis labels only. No model is called.
Three questions, one screenshot
The source is a 2400 by 1600 dashboard image. The chart caption is drawn in 28-pixel text, the bars of the two tallest quarters differ by 12 pixels, and the value above the Q3 bar is printed in 9-pixel text. An upstream OCR transcript of 120 tokens contains the caption and axis labels but not the bar heights or the small Q3 value. In this fixture text is readable at 6 pixels and bar heights are distinguishable at 3 pixels after resizing.
The opening request cannot be answered
The opening Question about the chart asks for the Q3 value and sends only the image, with Longest image side after resizing at 768 pixels. That is 768 by 512, one 768-pixel tile, 258 tokens, and the Q3 text shrinks to 2.88 pixels: unreadable. Set Input sent with the question to OCR transcript only and it costs 120 tokens but the value is not in the transcript. The question needs pixels at a readable scale.
Resolution, crop and cost
Without cropping, 1536 pixels still leaves the label at 5.76 pixels; 1664 pixels gives 1664 by 1109, six tiles, 1548 tokens and a 6.24-pixel label, which is answerable. Check Crop to the chart region before resizing: the chart is 1200 by 800, so at 1024 pixels the image is 1024 by 683, two tiles, 516 tokens and a 7.68-pixel label, answerable at a third of the cost. Sending the full 2400 by 1600 image costs twelve tiles, 3096 tokens.
Match the input to the question
Ask for the caption with OCR transcript only: it is answerable for 120 tokens, and no image is needed. Ask which quarter has the tallest bar: the transcript cannot answer it, an image at 256 pixels is 256 by 171, 258 tokens, with a bar gap of 1.28 pixels, too small; at 640 pixels the gap is 3.2 pixels and one 258-token tile answers it. Sending both transcript and image adds the 120 transcript tokens.
Reading the run and its limits
Play, Next event, Restart, Speed and Inspection time step through what is sent, the token count, the scaled detail sizes and the verdict over sixty seconds; the table repeats every number. The token rule is the tile rule documented for the Gemini models the linked article describes, and newer models and media-resolution settings count differently. Readability thresholds are fixture assumptions, not measured model behaviour, and no model is called.