Pub/sub system design architecture — topics, partitions, guarantees, DLQ

What publish/subscribe actually promises

Publish/subscribe is a delivery pattern in which a producer hands a message to a broker and then stops caring what happens next. It does not name a recipient, does not wait for one, and never learns whether anybody was listening. Subscribers register interest separately, are served separately, and fail separately. Almost everything interesting about the pattern, good and bad, is a consequence of that single property.

The promise is that a seventh consumer of an event can be added without editing, redeploying, or even notifying the service that emits it. The price is that the emitting service can no longer tell you who its callers are, and neither can your code search. A synchronous call graph is discoverable by reading imports and following types; a pub/sub graph exists only in broker configuration and in the memory of whoever wired it up.

This page treats pub/sub as an architectural pattern rather than as a product. The vocabulary below - topic, subscription, delivery semantics, retention, backlog - appears under different names in every broker, but the trade-offs are the same whether the substrate is a managed cloud bus, a partitioned log such as Kafka, a shard-based stream such as Kinesis, a rule-matching event router such as EventBridge, or a small footprint broker such as MQTT.

Reading the diagram - what each piece owns

The diagram names the moving parts, and it is worth being precise about which of them is a real boundary. A publisher owns serialization and the choice of destination; once the broker acknowledges the write, its involvement is over. A topic is a name plus a routing rule, and it is the only contract the publisher signs. Subscriptions are the piece most designs under-model: a subscription, not a consumer process, is the thing the broker actually tracks. It owns the cursor or the unacknowledged set, the redelivery counter, the filter, the retry policy, and the dead-letter destination. Two processes attached to one subscription divide the traffic between them; two subscriptions over the same topic each receive everything.

The middle row is the guarantee surface, and every item on it is scoped rather than absolute. Delivery semantics describe the broker-to-consumer hop and nothing beyond it. Ordering holds within a key or a lane, never across a whole topic. Retention governs the stored message, which routinely outlives every consumer that has already processed it. The lower rows are governance - the parts nobody builds on day one and everybody needs by the time a third team subscribes.

The one box that appears in no broker's documentation is the boundary itself: the shared understanding of what an event means and what its fields are allowed to do next quarter. That is the piece that breaks in practice, and the section on schema evolution further down is about defending it.

One publish, three subscribers - a concrete trace

A payments service emits a payment.captured event, roughly 2 kB of JSON, at about 1,200 events per second at peak. It is published to one topic and it has three subscribers, none of which the payments team can name from memory.

The ledger writer must never lose an event and needs per-account order, because a capture and a later partial refund applied out of sequence produce a wrong balance. It uses a durable pull subscription keyed on account id, a small redelivery budget, and a strictly idempotent write path. The fraud scorer wants the event inside a second and genuinely does not care about duplicates or order; it uses push delivery to an autoscaling endpoint and treats a redelivered event as a cheap re-score. The warehouse loader batches events into object storage every five minutes and is perfectly happy running six hours behind during a backfill; it uses a pull subscription with a long acknowledgement deadline and a large outstanding-message window.

Three consumers, one publish, three completely different configurations of the same guarantees. That is the shape of a healthy pub/sub deployment: the topic is a single, boring contract, and each subscription buys exactly the delivery properties its owner is willing to pay for. It is also where the failures come from - on the day the fraud endpoint starts returning 503s, only its subscription's backlog grows, and the payments team learns about it from a dashboard rather than from a stack trace in their own service.

The traceability you trade away

Decoupling is usually sold as a purely positive property, so it is worth stating the bill plainly. Once a producer no longer references its consumers, four things stop working. You cannot answer "who consumes this event" from source code. You cannot deprecate a field with confidence, because the set of readers is unknown. You cannot reconstruct a causal chain from a stack trace, because there is no stack that spans the broker. And you cannot reason about end-to-end latency by summing hops, because one of the hops is a queue whose depth is a function of unrelated traffic.

Three cheap habits recover most of what was lost. First, put a correlation id and a causation id in every envelope: the correlation id is constant for a whole business transaction, the causation id names the specific message that produced this one. With both, a chain that fans out through four topics can be reassembled by query rather than by guesswork. Second, propagate trace context through message attributes and join the consumer span to the producer span as a link rather than as a parent - publish and consume are not nested in time, and modelling them as parent and child produces spans that appear to last for the length of the backlog. Third, keep a registry of subscriptions with a named owning team, because the broker knows every subscriber and your organisation does not.

Topic-based routing versus content-based routing

Routing is where the pattern's cost model is decided. With topic-based routing, the producer picks the destination at publish time by writing to a name, and the subscriber selects whole names or name patterns. Matching is a lookup: cheap, predictable, and independent of message size. The cost lands on the namespace instead. Because the only way to let a subscriber narrow its intake is to give it a narrower name, topic hierarchies grow fast - orders.eu.retail.created rather than orders - and each new dimension multiplies the name count. Producers end up encoding routing decisions they should not have to know about, and adding a dimension later means republishing to new names.

With content-based routing, subscribers register predicates and the broker evaluates them per message. The namespace stays small and subscribers express precisely what they want, but the broker now does work proportional to the number of subscriptions on every single publish. A naive implementation is O(subscribers) per message and collapses at a few thousand subscriptions; real routers index the predicates instead, grouping subscriptions by the attributes they test so that one pass over a message's attributes narrows the candidate set before any full evaluation.

Why brokers filter on attributes and not payloads

Nearly every broker restricts filtering to a set of key-value attributes carried outside the body. That restriction is deliberate. Filtering on the payload forces the broker to deserialize it, which means the broker must know the schema, must be redeployed when the schema changes, and must pay parse cost per message per subscription rather than once. Attribute filtering keeps the body an opaque blob that the broker copies without inspecting. The practical rule: put in the topic name what you would shard on, put in attributes what subscribers would filter on, and put everything else in the body. Promoting a body field to an attribute later is a producer change that every existing subscriber survives; the reverse is not.

Fan-out is a write amplification problem

One publish becomes N deliveries, and the arithmetic is unforgiving. At 2 kB per message and 1,200 messages per second, ingest is about 2.4 MB/s. With six subscriptions the broker must move roughly 14.4 MB/s outbound, plus redeliveries, plus whatever replication factor the storage layer applies to the single stored copy. Doubling the subscriber count doubles egress and doubles the acknowledgement traffic coming back, while the publisher's bill and its dashboards do not move at all.

How the amplification lands depends on the storage model. A shared log stores the message once and gives each subscription a cursor over it: storage is O(1) in subscribers, egress is O(N), and adding a subscriber is nearly free until it starts reading. A copy-per-subscription broker materializes a separate queue per subscriber: storage becomes O(N), but each copy gets its own retention clock, its own redelivery counter, and its own dead-letter lane, which is why per-subscription failure isolation tends to be better on this model. Neither is wrong; they fail differently under the same load.

The subscription nobody owns

Unbounded fan-out is rarely a deliberate decision. It arrives as subscription sprawl: self-service creation, no owner field, no expiry. What an abandoned subscription then costs depends entirely on which retention rule the broker applies, and the two rules fail in opposite directions.

Under acknowledgement-gated retention - a copy-per-subscription queue, or a shared log that will not reclaim a message until every subscription has acknowledged it - a dead subscriber pins storage for everyone. Its backlog grows without bound, disk usage climbs with no change in publish rate, and the bill arrives long before anyone connects it to a subscription created months ago for an experiment. Under time or size retention, where the log discards segments on a clock regardless of any cursor, the trap is the mirror image: the abandoned subscription costs nothing, alarms on nothing, and is entirely invisible - until someone revives it and finds its stored position expired days ago. There is no error at creation time and no metric that moved; the consumer simply discovers that the messages it was going to resume from are gone, and it must choose between starting at the oldest surviving message or at the newest, both of which are wrong.

Same governance control fixes both signatures: every subscription carries an owning team and a review date, and any subscription with no acknowledgement activity for a fixed window gets alerted on and then deleted deliberately rather than discovered accidentally.

Delivery semantics, and why exactly-once is a pipeline property

There are only two things a broker can genuinely offer, and they differ by where the acknowledgement sits relative to the work. Acknowledge first, then process: if the consumer dies mid-work the message is already gone, which is at-most-once - no duplicates, occasional loss. Process first, then acknowledge: if the consumer dies after the side effect but before the ack, the message is redelivered, which is at-least-once - no loss, occasional duplicates. Every real system that cares about its data picks the second.

Exactly-once cannot be a property of the broker because the failure that produces duplicates happens outside it. The acknowledgement travels over a network that can drop it after the effect has already been committed, and no protocol removes that window; it can only be moved. Broker features marketed as exactly-once are deduplication over a bounded window - a publisher sequence number, a session, a retention period - and they are genuinely useful, but a redelivery that arrives after the window closes is an ordinary duplicate again.

What actually delivers exactly-once effects is the consumer being unable to apply the same message twice: a deduplication key checked in the same transaction as the write, a conditional update, or an operation that is naturally idempotent. That machinery has its own page - see idempotency architecture for the dedupe store and key design, and exactly-once semantics in streaming for the transactional variant. The design rule for pub/sub is short: assume at-least-once at every subscription and make duplicate handling a consumer requirement, not a broker setting.

Push and pull are a flow-control decision

The delivery direction looks like a transport detail and is actually the answer to "who decides how fast this consumer works".

With pull, the consumer asks for messages when it has capacity, so the arrival rate is bounded by the consumer's own loop. Overload is impossible by construction: excess work sits in the broker's backlog, which is exactly what durable storage is for. The costs are latency and waste - naive polling either burns requests on empty responses or adds a poll interval to every message's delivery time, which is why every serious client uses long polling or a persistent stream with a credit window instead of a fixed poll cycle.

With push, the broker delivers as fast as it can and the consumer must be an addressable endpoint. Latency is lower, there is no idle polling, and serverless consumers can scale from zero. But the consumer no longer controls its own arrival rate, so overload becomes possible and the only brake is the failure signal: the consumer must reject with a status the broker treats as retryable, and the broker must implement backoff with jitter. Push also drags a whole second contract along - TLS, authentication of the broker to the endpoint, and an autoscaler tuned on request rate rather than on backlog. General backpressure mechanics are covered separately; the pub/sub-specific point is that choosing push moves flow control from the consumer to the broker, and that decision is not reversible without redeploying the consumer.

The acknowledgement deadline is a lease

Whichever direction is used, an in-flight message is leased rather than removed. The broker hands it out, starts a timer, and redelivers if no acknowledgement arrives before the timer expires. This makes the deadline a real design parameter. Set it below the consumer's p99 processing time and slow-but-successful work is redelivered while the original is still running, producing duplicate effects and a load spiral that looks like a consumer bug. Set it far above and a crashed consumer's messages sit invisible for that long before anyone else can take them. The usual answer is a deadline near p99 plus a client that extends the lease while work is genuinely in progress - and if extension is happening on most messages, the work belongs in a job store with the message as a pointer, not in the message handler.

Subscription models: shared lane, per-subscriber copy, consumer group

Three arrangements get called "subscribing" and they are not interchangeable.

A shared lane gives every attached worker a slice of the same message set: each message is processed once, and adding workers adds throughput. This is the competing-consumers arrangement familiar from work queues. A per-subscriber copy gives every registered subscriber the whole message set - this is fan-out proper, and it is the arrangement that makes pub/sub different from a queue. A consumer group is the composition of the two: the group is one logical subscriber that receives everything, and its members divide the group's share among themselves. Group membership changes trigger a reassignment of work, which has its own failure modes - see consumer rebalancing.

The classic mistake is at the seam between the second and third. A team scales a consumer by deploying a second instance, but the deployment creates its own subscription rather than joining the existing one. Every message is now processed twice by identical code, and because both instances succeed, nothing alarms. The symptom is duplicated side effects with no errors anywhere - doubled emails, doubled ledger entries - and it is found by inspecting subscription counts, not logs.

The second decision is durability of the subscription itself. A durable subscription accumulates messages while its consumer is offline, so a deploy costs backlog rather than data. An ephemeral one only receives what arrives while it is connected, which is right for live dashboards and catastrophic for anything that must reconcile.

Ordering guarantees have a scope, and the scope is the whole answer

Total order across a topic means every message passes through one serialized lane, which caps throughput at what a single writer and a single consumer can sustain. Almost nobody needs it. What applications actually need is order within an entity: this account's events in sequence, this device's readings in sequence. So brokers offer order scoped to a key, and the key selection is simultaneously the parallelism decision. Choose a key with too little cardinality and the lanes are lopsided; choose a key that does not match the invariant - order per shipment when the invariant is per order - and the guarantee is technically upheld and practically useless.

Two scope limits catch people out. Order is a property of one subscription, so two subscribers of the same topic can observe the same messages at wildly different positions; "the ledger has already seen it, therefore the search index has" is never a valid inference. And there is no ordering across topics at all: if a workflow depends on order.created being handled before payment.captured, and those are separate topics, the sequence must be enforced by the consumer, usually by keeping state and parking the early arrival.

Strict order and dead-lettering pull against each other

Ordered delivery means a failing message blocks its lane, because skipping it is by definition a reordering. That leaves two choices and no third. Block - the lane stalls until the message succeeds or an operator intervenes, preserving correctness and sacrificing liveness for every entity sharing that lane. Or skip - route the message aside and continue, preserving throughput and admitting a permanent gap in a sequence the consumer believed was complete. Systems that choose to skip need the consumer to detect the gap, which means sequence numbers in the envelope; otherwise the consumer silently applies event 5 to state that never saw event 4.

Retention turns a bus into something you can rewind

Retention is how long the broker keeps a message, and there are two distinct policies wearing the same word. In the delete-on-acknowledge model, retention is a safety net for undelivered messages: once every subscription has acknowledged, the message is gone and the past is unrecoverable. In the retained log model, messages live for a fixed window regardless of consumption, and a subscription is just a cursor - so replay is repositioning that cursor rather than republishing anything.

The retained log buys three capabilities that are hard to get any other way. A new consumer can bootstrap itself from history instead of requiring a separate backfill pipeline. A consumer that processed a window of messages incorrectly can reprocess it after the fix. And a derived store - a cache, a search index, a read model - can be dropped and rebuilt from the log, which changes it from a database that must be migrated into a projection that can be regenerated. That last property is the bridge to event sourcing and CQRS, which take it further than a message bus does.

Replay is only safe if the consumer is replay-safe

Replay re-emits effects. Reprocessing a day of payment.captured through a consumer that sends receipt emails sends every receipt again, and the deduplication key is the only thing standing between a fix and an incident. Replay also competes for capacity: pushing six hours of backlog through the same subscription that serves live traffic makes live traffic wait, so replay belongs on a separate subscription or consumer group with its own throughput budget. And replay runs at whatever rate the broker can deliver, which is frequently a hundred times production rate - fast enough to trip every downstream rate limit the consumer normally never approaches. Throttle the replay deliberately.

Two sizing rules follow. Retention must exceed the worst realistic consumer outage plus the time it takes to notice one, or an incident silently becomes data loss. And whatever the retention window is, the consumer must be able to decode every message format written during it - which is what makes schema evolution a retention problem as well as a deployment problem.

Backlog is the signal; almost everything else lags

Two numbers describe the health of a subscription, and only one of them makes a good alert. Backlog depth - undelivered or unacknowledged message count - is throughput-dependent: 50,000 messages is four seconds of work for one consumer and four hours for another, so a fixed threshold means something different on every subscription. Age of the oldest unacknowledged message is already in the unit the business cares about, is comparable across subscriptions, and maps directly onto a freshness objective. Alert on age; graph depth for capacity work.

The derivative is what turns a dashboard into a diagnosis. If arrival rate exceeds service rate, backlog grows without bound and the time to drain once service recovers is roughly the accumulated backlog divided by the surplus service rate - which is why a ten-minute outage of a consumer running at 90% utilisation takes far longer than ten minutes to clear. Two thresholds are worth wiring in every deployment: alert when age of oldest exceeds the freshness objective, and page when it exceeds a quarter of the retention window, because that is the point at which a lag problem is turning into a loss problem.

Report these per subscription, never per topic. A topic-level dashboard averages a healthy consumer against a stalled one and shows green while a subscriber is four hours behind. Publish-side signals are worth a panel too: rejected or throttled publishes, and publish latency, which is the first place a broker under storage pressure shows itself.

# the four panels worth having per subscription
oldest_unacked_age_seconds     # alert > freshness SLO; page > retention/4
backlog_message_count          # capacity planning, not alerting
delivery_attempts_total        # rising ratio to acks = retry storm forming
dead_letter_count              # any sustained non-zero rate needs an owner

Poison messages, retry storms, and the dead-letter lane

A message that can never be processed successfully will be retried forever unless something counts. The counter belongs to the subscription, and when it is exhausted the message moves to a dead-letter destination with its failure metadata attached. The mechanics of retry budgets, quarantine, triage, and redrive have their own page - see dead-letter queue architecture - so what matters here is the part that is specific to fan-out.

Under fan-out, poison is a per-subscriber property, not a property of the message. An event with an unfamiliar enum value may be unparseable to the ledger consumer and entirely fine for the analytics loader. That means dead-letter destinations must be per subscription: a shared dead-letter topic loses the one fact triage needs, namely which subscriber failed. It also gives you a free diagnostic - the same message id appearing in several subscriptions' dead-letter lanes points at the producer, while one lane's worth points at that consumer.

Retry storms are the other half. When a downstream dependency degrades, every in-flight message across every subscription fails at roughly the same moment and is redelivered together, so the load offered to the struggling dependency spikes exactly when it can least absorb it. Exponential backoff alone does not fix this, because synchronized failures produce synchronized retries; the backoff needs jitter, a cap on concurrent in-flight work per consumer, and ideally a circuit breaker that stops pulling entirely while the dependency is down. Stopping consumption during an outage is the correct behaviour: the backlog is durable storage doing its job.

Schema evolution across an anonymous boundary

This is where pub/sub systems break most often, and the reason is structural rather than technical. In a synchronous API the provider can enumerate its callers, measure their traffic, and coordinate a change. Across a topic the producer does not know who is reading, so "is this change safe" has no answerable form. The producer and every consumer deploy on independent schedules, which means that for some window both the old and the new shape of the event are in flight simultaneously. Any change that is not tolerable in both directions during that window is an outage waiting for a deploy.

The discipline that survives this is narrow. Add fields, never remove or rename them. Give every added field a default so a reader that has not been updated still constructs a valid object. Never repurpose an existing field, never narrow a type, and never tighten a validation rule, because all three break readers that were correct yesterday. On the reading side, be a tolerant reader: ignore unknown fields instead of rejecting them, and treat a missing optional as its default rather than as an error. A genuinely breaking change is not a change at all - it is a new event type, published alongside the old one until every consumer has migrated, after which the old type is retired. Because the consumer set is unknown, dual publishing is the only migration that does not require knowing who to ask. The compatibility rules and the tooling that enforces them at registration time are covered in schema registry architecture.

The envelope is a separate contract from the payload

Keeping a small, stable envelope around a versioned payload solves several problems at once. The envelope carries what the infrastructure needs - id, type, version, timestamp, source, correlation and causation ids, and an optional sequence number - and it changes almost never, so routing, filtering, tracing, and deduplication all work without deserializing the body. The payload changes at whatever pace the domain requires.

{
  "id": "01J9Z8...",              // unique; the consumer's dedupe key
  "type": "payment.captured",     // routing and dispatch, never inferred from shape
  "version": 3,                   // payload version, not envelope version
  "occurred_at": "2026-07-06T11:04:18Z",
  "source": "payments-api",
  "correlation_id": "chk_88213",  // constant across the whole transaction
  "causation_id": "01J9Z7...",    // the message that caused this one
  "sequence": 42,                 // per-key; lets a consumer detect a gap
  "data": { }                     // the only part allowed to change often
}

Retention makes old versions immortal. If the log holds thirty days and three payload versions were written during them, a consumer replaying from the start must decode all three. Version handling is therefore not transitional code to be deleted after the rollout - it lives as long as the retention window, and deleting it early turns a routine replay into a failed one.

Failure modes worth recognising on sight

Most pub/sub incidents are one of a handful of shapes, and each has a signature that identifies it before the logs do.

The last row is the one that has no quick fix and the largest long-term cost. Every other failure here is visible in a metric; losing the ability to reason about cause and effect shows up only as a slow decline in how confidently anyone can change the system.

When pub/sub is the wrong shape

The pattern is a poor fit whenever the producer actually needs something back. If the caller cannot proceed without the result, a request/response call is simpler, faster, and far easier to debug than a topic plus a reply topic plus a correlation table. Pub/sub is also wrong when total order at high throughput is a real requirement, because those two demands are directly opposed and no configuration reconciles them.

It is wrong, or at least insufficient, when the publish must be atomic with a database write. Writing to the database and then publishing is two operations with a crash window between them, which loses events; publishing first loses the opposite way. That specific problem is solved by the transactional outbox, which turns the publish into a row in the same transaction. Multi-step workflows that need compensation on failure want an explicit saga rather than a chain of topics, because a chain of topics has no way to express "undo the first three steps".

Finally, it is wrong when the value of the event does not cover the fan-out cost - very large payloads to many subscribers are usually better published as a small notification carrying a pointer, with subscribers fetching the body only if they care. And if you truly need to know exactly who is consuming what, the anonymity that makes the pattern valuable is working against you; that is an argument for explicit interfaces, not for a bigger broker.