MULTIMODAL · TIME ALIGNMENT
Multimodal agent: align frames before describing an event
Sample video frames, place the audio event on the same clock, and see whether the description keeps cause before effect.
Event trace
Computed data
Final metrics
A fictional six-second clip: the cup falls between 3,200 and 3,600 ms and the crash is recorded at 3,500 ms. Frames are described by fixed labels, not by a vision model. Out-of-order delivery swaps the first shards frame with its predecessor. No Gemini or ADK media API is used.
One clock for frames and sound
The fictional clip is six seconds long. A cup tips at 3,200 ms, is visibly falling until 3,600 ms, and the crash is recorded in the audio at 3,500 ms. The describer only sees sampled frames and the audio event, placed on the same clock. A correct description needs a frame inside the fall, frames in their true order, and a crash paired with a frame showing the fall or its aftermath. In the opening run, one frame per second samples 0, 1,000 ... 5,000 ms: the cup is on the table at 3,000 ms and in shards at 4,000 ms, and the fall itself is never seen.
Sampling rate and keyframes
Set Frame sampling rate to 2 frames per second. The frame at 3,500 ms falls inside the fall window and the crash pairs with it exactly, so the description is correct. Three frames per second catch 3,333 ms, 167 ms from the crash. Half a frame per second samples only 0, 2,000 and 4,000 ms. Return to one frame per second and enable Add a frame at the audio event time: an extra frame at 3,500 ms rescues the description with seven frames instead of twelve.
An audio offset moves the cause
At two frames per second, set Audio track offset to -800 ms. The crash is placed at 2,700 ms, and the nearest frame, 200 ms away at 2,500 ms, shows the cup still on the table. Within the 250 ms Pairing tolerance the sound is attributed to that frame, so the description puts the effect before its cause. Lower the tolerance to 100 ms and the crash is instead left unexplained. An offset of +400 ms pairs the crash with the shards frame at 4,000 ms, which is still a correct story.
Delivery order
Frame delivery can reorder the first shards frame ahead of its predecessor. Out of order, re-sorted by timestamp restores the true order and the description stays correct. Out of order, no timestamps uses arrival order, so shards appear before the cup falls and the story is reversed. Play, Next event, Restart, Speed and Inspection time step through sampling, audio placement, delivery, the fall window and pairing over sixty seconds.
What the model leaves out
Frames carry fixed labels instead of being interpreted by a vision model, and the audio has a single labeled event. Real multimodal inputs can drift over time, have variable frame intervals and contain several overlapping events. Many systems sample video at about one frame per second, which is exactly the regime in which short events are missed. No Gemini or ADK media API is used.