ADK multimodal — images, audio, documents as first-class inputs

Everything the model sees is a Part

ADK has no separate image API bolted onto a text API. It has one content model, borrowed from the underlying google-genai types, and every modality lives inside it. A message is a Content with a role and a list of Part objects, and a Part is a tagged union: it carries either text, or binary data, or a reference to data stored elsewhere, or a function call, or a function response. Nothing else. Once that lands, ‘sending an image’ is just appending a differently-shaped part to a list you were already building.

Three shapes matter for media. part.text is the ordinary string. part.inline_data is a Blob — a mime_type plus raw data bytes — carried inside the request. part.file_data is a FileData — a mime_type plus a file_uri — carried as a pointer the model service dereferences on its own side. One turn can mix all of them:

from google.genai import types

message = types.Content(
    role="user",
    parts=[
        types.Part(text="Which line item is the delivery fee?"),
        types.Part(inline_data=types.Blob(
            mime_type="image/jpeg", data=jpeg_bytes)),
        types.Part(file_data=types.FileData(
            mime_type="application/pdf",
            file_uri="gs://receipts/invoice-8812.pdf")),
    ],
)

Order is not cosmetic: the model reads parts in sequence, so a question placed before several images is answered with all of them in view.

Three ways to hand over bytes

Given the same photo you have three genuinely different transport options, and picking wrong causes both cost surprises and hard errors. Inline puts the bytes in the request: no upload step, no cleanup, bounded by the total request size the endpoint accepts — tens of megabytes including base64 expansion, smaller than it sounds once you attach a second file. Use it for screenshots and camera captures, anything a user just typed a message about.

File URI uploads the bytes once to a store the model can reach — the provider’s file service, or a Cloud Storage bucket on Vertex — and references them by URI. That clears the request ceiling and is far cheaper when one document is referenced across many turns: you upload the fifty-page contract once, not on every request. The cost is lifecycle — uploaded files expire, and a stale URI fails at request time, not attach time. Artifacts are ADK’s own layer over all this, answering a different question: not ‘how does the model reach the bytes’ but ‘where do they live between turns, and who owns them.’

Artifacts — where media lives between turns

An ADK artifact is a named, versioned binary belonging to a session or a user, held by an ArtifactService you configure on the Runner just as you configure a SessionService — in-memory for development, Cloud Storage—backed in production. The unit it stores is a Part, the same type the model consumes, so an artifact is not a side-channel file; it is model content you have parked.

Tool and callback contexts expose the save and load operations, and two properties shape how you use them. Saves are versioned: writing the same filename appends a version rather than overwriting, so chart.png can be regenerated across a conversation with every earlier version still addressable. Filenames are also namespaced: a plain name is scoped to the session and dies with it, while the conventional user: prefix promotes the artifact to the user and makes it visible from every future session — the difference between ‘the image they just uploaded’ and ‘their ID document on file.’

The discipline that matters is that an artifact is not in the context. Saving a PDF costs zero tokens and keeps costing zero until something loads it into a request. ADK ships a built-in tool that lets the model pull artifacts in by name when it decides it needs to look; the alternative is loading them deterministically in a callback. Either way, media enters the context just in time.

MIME types are the contract

The mime_type on a blob or file reference is not metadata — it is the instruction that decides how the bytes are decoded. Get it wrong and you do not get a helpful message about image formats; you get a decode error, or a confident answer about nothing. Declare what the bytes are, never what the filename claims.

Two traps recur. DOCX, XLSX and PPTX arrive constantly, are ZIP containers rather than documents, and passed through as opaque blobs produce gibberish — convert to PDF at the door. And double encoding: the blob field wants raw bytes, so code that base64-encodes first hands the model an ASCII string it tries to read as a JPEG.

What media actually costs

Media is the biggest source of context bloat in an agent, and the arithmetic is worth memorising because it is unintuitive. A small image costs roughly what a page of prose costs. A minute of audio costs a couple of thousand tokens. A minute of video costs an order of magnitude more than a minute of audio, because you pay for sampled frames and the soundtrack. A PDF page costs an image plus the text on it.

The published figures for the current Gemini generation sharpen that: an image small enough to fit one tile bills as a flat couple of hundred tokens, and larger images are cut into tiles that each bill the same amount — cost scales with area, not with file size. Audio runs at a few tens of tokens per second; video at a few hundred per second at default resolution, with a low-resolution mode cutting that by roughly four. Treat the constants as a shape rather than a promise and check them for the model you deploy.

Two consequences follow. Uploading a 12-megapixel phone photo is waste — past the resolution the model tiles at, extra pixels buy tokens rather than detail; crop to the region if you need fine print. And media is sticky: an image attached at turn three sits in the history at turn twenty, billed again on every request in between.

Documents — a PDF is read as pages, not as text

Native document understanding deletes the most fragile pipeline you own. The model does not receive your PDF as an extracted text dump; it receives the pages and reads them the way a person does — columns as columns, a table as a grid, handwriting in the margin as handwriting. Everything a text extractor silently destroys about a form is exactly what the model needs to fill it in.

The envelope is generous but real: current models accept documents in the high hundreds of pages, and each page bills as an image plus its text. Large limit, per-page cost — that combination drives the design. If a question concerns one clause of a two-hundred-page contract, sending all two hundred pages is economically foolish; keep the document as an artifact, maintain a cheap text index or per-section summary beside it, and load only the relevant page range.

Two habits sharpen accuracy. Ask for structure, not prose — pair the document with a response schema or explicit field list, because ‘extract the invoice details’ returns a paragraph you must then parse, reintroducing the brittleness you removed. And ask for provenance: a required page number beside every extracted value turns an unverifiable claim into a checkable one.

Video — sampled, not watched

Video is the modality whose mental model most needs correcting. The model does not watch your file; the service samples it into stills at a low fixed rate — on the order of one frame per second — extracts the audio track, and reasons over that sequence. Motion faster than the sampling rate is therefore invisible: a ball crossing the frame in 300 ms may not appear at all, and ‘how many times did the light blink’ asks a question the representation cannot answer. The audio track, by contrast, is continuous, so a lecture is understood far better than a magic trick.

Because frames are billed, duration is the cost variable, and both levers are about sending less. Most services accept a clipping interval — a start and end offset — so a question about one moment in an hour-long recording sends ninety seconds. A low-resolution setting is the right default whenever you need to know what is happening rather than to read text on screen. One property comes free: frames are timestamped, so the model answers ‘when’ questions with offsets you can seek to.

Audio files, and where the voice agent takes over

Audio enters an ADK agent in two entirely different ways, and conflating them causes real confusion. This article covers the first: an audio file — a voicemail, a recorded meeting — attached to a normal turn as a blob or a file reference. The second is live audio, where PCM frames stream continuously into a LiveRequestQueue under run_live — a real-time system with its own format constraints, latency budget and barge-in problem, and the voice companion’s territory.

For files, the thing worth understanding is that native audio comprehension is not transcription with extra steps. Transcription is one thing you can ask for; the model also hears what a transcript deletes — who was speaking, whether they sounded frustrated, that two people talked over each other, that eight seconds of silence preceded the answer. ‘Summarise the complaint and flag whether the caller became angry’ is one request, not an ASR pass plus sentiment analysis over its output. Design around duration: chunk multi-hour recordings and carry per-segment summaries forward rather than the audio.

Media coming back from a tool

Agents do not only consume media users send; tools produce it — a chart, a screenshot, a generated PDF. The awkwardness is that an ADK function tool returns a dictionary, serialised into a function-response part, and megabytes of base64 in that dictionary is wrong twice over: it bloats the response the model must read, and it is not typed as an image, so the model cannot look at it.

Split the payload from the announcement. The tool writes the bytes to the artifact store and returns a small, honest dictionary describing what it made; the media reaches the model either because the model calls the built-in artifact-loading tool by name, or because a callback loads it deterministically before the next model call.

async def plot_order_history(user_id: str, tool_context) -> dict:
    png = render_chart(fetch_orders(user_id))          # raw bytes
    part = types.Part(inline_data=types.Blob(
        mime_type="image/png", data=png))
    version = await tool_context.save_artifact("order_history.png", part)
    return {"status": "ok",
            "artifact": "order_history.png",
            "version": version,
            "summary": "12 orders, spend rising since March"}

The summary field is not decoration. It lets the agent answer cheaply when nobody needs to look at the picture, so the image is loaded only when a question genuinely requires vision.

Generated media works the same way. An image model returns response parts carrying inline image data — the part shape coming back is the part shape going in — so you save it as an artifact and everything above applies unchanged. Spoken output is the exception, because it is a property of the run rather than of a part: response modalities and voice are selected on the run configuration, and that machinery belongs to the voice-agent article.

Failure modes that actually bite

Multimodal bugs are unusually opaque, because the thing that failed is bytes and the error message is about a request. Roughly by frequency:

The first costs the most engineering time, because it does not look like a media problem. Capability is per-model, not per-family: a lightweight variant may accept images but not video, and live-API models constrain which input modalities they take at all. Check the MIME type against what your configured model accepts before building the request, so a user gets ‘I can’t read video’ not a stack trace.

Keeping media out of the context you keep paying for

The place to enforce all of this is a callback — the one hook that sees the request after the agent assembles it and before it costs anything. A before_model_callback can rewrite the outgoing request: downscale an oversized image, reject an unsupported type, redact a document, or — the highest-value edit — walk back through the history replacing media parts older than the current turn with a text stand-in such as [image: damaged package photo, reviewed at turn 3].

That one transformation is what keeps a long multimodal conversation affordable. The model almost never needs to re-examine a photo it already described; it needs to remember that a photo was seen and what was concluded. Swapping pixels for that sentence turns a cost that compounds every turn into one that does not. Pair it with the obvious hygiene: a size cap and type allowlist at upload, and media-token logging beside your text tokens.

Finally, evaluate on the media you will actually receive. A receipt agent that scores well on clean sample PDFs will be met with blurry phone photos of faded thermal paper, shot at an angle, with a handwritten tip. Those failures are modality-specific and invisible to a text test suite; build the eval set out of real, ugly inputs.