Why architecture matters here
Message systems fail on subtle guarantees. "At-least-once" is fine until a duplicate corrupts state. "Ordered" is fine until an ordering key creates a hot partition. "Push" is fine until your service cannot keep up. The architecture matters because every combination of these decisions has a name, a set of trade-offs, and a matching consumer pattern.
Pub/Sub gives you the knobs; picking them well is the design work. And once designed, keeping the pipeline healthy means watching backlog age, DLQ growth, and outstanding-message counts.
With the architecture in your head you can plan schemas, retention, ack deadlines, and consumer parallelism without surprises.
The architecture: every piece explained
The top strip is the publish + subscribe path. Publisher batches messages (up to N messages or M bytes) and attaches optional ordering keys. Topic is global metadata — a name and schema — that spans regions. Subscription is a regional queue view over that topic; multiple subscriptions on the same topic get independent copies of the messages. Delivery mode is push (Pub/Sub POSTs to your endpoint), pull (client polls), or stream pull (long-lived gRPC stream).
The middle row is the guarantees. Ack window is the per-message deadline within which the consumer must acknowledge; missing it triggers redelivery. Message ordering preserves order per ordering key — different keys can be reordered against each other, which is how you get parallelism without losing per-entity order. DLQ topic receives messages that fail delivery beyond a retry threshold. Exactly-once uses server-side dedupe within a window so the same message is not delivered twice to a well-behaved consumer.
The lower rows are operations. Flow control lets clients bound outstanding messages and bytes so a slow consumer does not OOM. Observability exports backlog message count, oldest unacked age, and DLQ counts — the vital signs. Consumer resilience is the client-side pattern: idempotency by message ID, controlled retries, and awareness of retention (default 7 days).
End-to-end flow
End-to-end: an order service publishes an event with ordering key = user_id. The publisher batches five events and sends to the topic. Pub/Sub durably stores the batch and fans it out to two subscriptions: analytics and fulfillment. The analytics subscription uses stream pull; a consumer receives messages, processes each into BigQuery, and acks. The fulfillment subscription uses push to a Cloud Run service; Pub/Sub POSTs with a signed token. The Cloud Run service is slow this hour; ack deadlines miss on a few messages; Pub/Sub redelivers. After three attempts, the problematic messages route to the DLQ topic. An oncall engineer inspects the DLQ, finds a schema bug in the consumer, deploys a fix, and replays the DLQ using a small script that publishes back to the main topic. Ordering is preserved per user_id throughout.