Why architecture matters here
Dataflow fails on cost surprises, wrong worker sizing, and streaming engine misuse. Architecture matters because autoscale + shuffle + windowing decide behavior.
The architecture: every piece explained
The top strip is the runtime. Beam pipeline SDK. Dataflow runner translate + execute. Streaming engine — offload state. Autoscaler workers.
The middle row is streaming. Shuffle service managed. Watermark + timers event time. Prime containers — Flex Templates. Snapshot / update in-place.
The lower rows are ops. IO connectors — Pub/Sub + BQ + GCS. Metrics + jobs UI. Ops — cost + drain + templates.
End-to-end flow
End-to-end: Beam pipeline reads Pub/Sub, windows events, writes to BQ. Dataflow runner picks streaming engine. Autoscale adjusts to 20 workers. Watermark + timers handle late data. In-place update deploys new version.