Message Queues — Kafka, RabbitMQ, SQS, and When to Use Each

Why it matters

Wrong queue choice locks in architecture for years. Kafka for a workflow that needs RabbitMQ semantics is painful; SQS for a use case requiring replay is worse. Learning the choices is core system design.

The architecture

Kafka: append-only log partitioned across brokers. Consumers track their own offset. Consumers can replay from any point. Optimized for throughput (millions of msg/s), retention (days to weeks).

RabbitMQ: broker manages message state, routes messages to consumers, acks remove from broker. Rich routing (topic exchanges, headers). Lower throughput but richer semantics.

One message, one worker - the pattern that defines a queue

A queue and a topic differ in exactly one place: what the broker does when two consumers are attached. A topic hands each of them a copy. A queue hands each of them a share. That single behavioural difference — competing consumers rather than fan-out — is what every other property below follows from, and it is why a queue is the natural home for work while a topic is the natural home for facts. The publish/subscribe side of the split, including routing models, subscription types and the cost of fan-out, is developed in pub/sub system design; this article stays on the work-distribution side and on the broker comparison the title promises.

The distinction shows up in payload naming before it shows up in any diagram. Queue messages are imperative and owned: resize-image, charge-invoice, send-receipt. Exactly one worker is supposed to act, and if two act you have a bug with a customer-visible name — a double charge, two shipping labels, two emails. Topic messages are declarative and unowned: InvoicePaid happened, and it is nobody's business how many services care. Teams that put imperative work on a fan-out topic find this out the first time somebody attaches a second subscriber "just to observe".

Work distribution also scales along a different axis from fan-out. Adding a worker to a queue adds throughput until a shared downstream saturates — a connection pool, a third-party API quota, a row that every job updates — at which point the queue quietly converts your extra workers into extra contention. Adding a subscriber to a topic adds load to the broker instead, because the broker must materialise another copy. The first is a capacity problem you diagnose by looking at the consumer; the second is one you diagnose by looking at the broker. Confusing them is how a team ends up scaling the wrong tier for a week.

The queue as a load-levelling buffer

The second thing a queue buys is time. A synchronous call demands that the callee have capacity at the instant of the call. A queue demands only that it have capacity eventually. That converts a provisioning problem into a scheduling problem, and the conversion is worth doing arithmetic on rather than asserting.

Take a checkout path that normally emits 400 events per second. A flash sale drives it to 5,000 for ninety seconds. The downstream fleet sustains 1,200 per second. Called synchronously, 3,800 requests per second get timeouts or 503s for a minute and a half, and the retry storm that follows makes it worse. Through a queue, the same burst deposits roughly 342,000 messages — the 3,800 per second excess across ninety seconds — and then drains at the 800 per second of spare capacity available once arrivals fall back to normal, so the backlog clears in about seven minutes. Nothing failed. What changed is the shape of the promise: instead of "this succeeds right now or not at all", you have promised "this succeeds within about seven minutes at the worst point of a burst". If the business can live with that, the queue is free capacity. If a customer is watching a spinner waiting for the result, a queue in front of that work solves nothing.

The same arithmetic exposes the limit that gets forgotten in design reviews. A queue absorbs variance, not deficit. If mean arrival rate exceeds mean service rate, the backlog grows without bound and the only endings are exhausted retention, exhausted broker storage, or an operator purging the queue by hand at 3am. Every alert worth having is really a test of that inequality — not "is the queue non-empty" but "is the oldest message getting older". Age of the oldest unprocessed message is the metric that means something; depth alone lies in both directions, since 200,000 messages draining in twenty seconds is healthy and 400 messages that have been sitting for an hour is an outage.

The bounded case deserves a deliberate decision rather than a default. An unbounded queue on a memory-backed broker turns a consumer outage into a broker outage, which is why RabbitMQ's memory and disk watermarks block publishers before that happens — and blocking publishers is backpressure whether or not anyone designed for it. Setting an explicit maximum length with an overflow policy at least makes the choice legible: reject at enqueue, drop the oldest, or dead-letter the overflow. Rejecting at the door is load shedding; the flow-control mechanics themselves are covered in streaming backpressure.

The lease is what makes at-least-once work

A queue broker maintains a per-message state machine: ready, then in flight once a consumer takes it, then gone when the consumer acknowledges — or back to ready if it does not. That in-flight state is a lease, and it is the whole crash-recovery mechanism. A worker that dies mid-job never acknowledges, the lease lapses, and the message becomes somebody else's problem. It is also the reason at-least-once is the default everywhere: the broker cannot tell a worker that died before doing the work from one that died after doing it and before acknowledging, so it must assume the worse case and hand the message out again.

What differs between brokers is the clock the lease runs on, and the differences are larger than the shared vocabulary suggests.

Three brokers, three different leases

SQS runs the lease on a wall clock: the visibility timeout, 30 seconds by default, started on receive and extendable per message while work is in progress. The mechanics and tuning of that timer, plus the redrive counter that rides along with it, are covered in Amazon SQS architecture.

RabbitMQ classically ran no timer at all. An unacknowledged message stays out for as long as the consumer's channel is open, and is requeued when the channel or connection closes. The lease is therefore a TCP liveness question, which means a consumer wedged in an infinite loop or a multi-minute stop-the-world pause keeps holding its messages while looking perfectly healthy to the broker. Recent versions add a blunt safety net — a consumer acknowledgement timeout, 30 minutes by default, which closes the channel rather than releasing the one message — but the operational model is still "connection health is message health", and client heartbeats are what actually detect a dead worker.

Kafka has no per-message lease at all. This is the largest single difference in this article and the one most often glossed over. Nothing on the broker knows that a record is being processed; there is no in-flight state to expire. What a consumer holds is an assignment of whole partitions, kept alive by two independent clocks — a background heartbeat thread governed by session.timeout.ms, and the requirement that your own code return to poll() within max.poll.interval.ms. Miss the second and the consumer is declared dead, its partitions are reassigned, and everything since its last committed offset is delivered again to whichever instance inherits them.

What that asymmetry costs when processing is slow

Picture one message that takes four minutes while the rest take two seconds — an unusually large file, a downstream API having a bad day. On SQS or RabbitMQ, the blast radius is that one message: it is redelivered, it may be processed twice, and an idempotent consumer absorbs the duplicate. On Kafka, with a five-minute poll interval and a fetched batch of 500 records, the slow record consumes the budget for the entire batch, the consumer misses its deadline, the group rebalances, every consumer in the group stops briefly, and the whole uncommitted offset range — including hundreds of records already finished — is reprocessed elsewhere. One slow message costs one message in a queue broker, and costs a partition plus a group-wide pause in a log broker. How the reassignment itself behaves is covered in consumer rebalancing.

The asymmetry drives two genuinely different remedies. On a queue broker you extend the lease on the individual message while work is demonstrably progressing. On a log broker you cannot, so you shrink max.poll.records until a batch reliably fits inside the interval, or you decouple entirely: hand records to a bounded worker pool, call pause() on the assigned partitions while that pool is saturated, keep calling poll() so the group still considers you alive, and commit only offsets whose work has actually completed. Either way redelivery produces duplicates, and duplicates are harmless only if the consumer is idempotent — see idempotency architecture.

In-flight limits and prefetch

Every queue client buffers ahead of the worker, and how much it buffers is a fairness-versus-throughput dial that most teams set once and never revisit.

RabbitMQ calls it basic.qos prefetch: the maximum number of unacknowledged messages the broker will push down a channel. At prefetch=1 a consumer holds only the message it is working on, so a slow worker can never hoard and dispatch is perfectly fair — at the cost of a full broker round trip of idle time between every message, which for two-millisecond jobs can halve effective throughput. At prefetch=500 the worker never waits, but five hundred messages sit in that consumer's private buffer where no other consumer can reach them, and if it stalls they stay invisible until the channel breaks. This is head-of-line blocking of a purely local kind, and it has nothing to do with ordering guarantees. The rough rule: keep prefetch small (1 to 5) when jobs take seconds and vary in duration, large (hundreds) when jobs are uniform and take milliseconds, and always small enough that prefetch times message size times consumer count fits comfortably in memory.

SQS expresses the same dial as two hard limits rather than a knob: at most ten messages per receive call, and a ceiling on messages in flight per queue — 120,000 for standard queues, 20,000 for FIFO. Hitting the in-flight ceiling is a genuinely confusing production event, because receives begin returning nothing while the queue is visibly deep. The fleet is holding the maximum number of leases, and the cause is almost always a visibility timeout set far longer than actual processing time, so completed work keeps its lease alive for minutes after it finished.

RabbitMQ   channel.basic_qos(prefetch_count=5)        # unacked ceiling per channel
SQS        ReceiveMessage MaxNumberOfMessages=10      # hard cap; in-flight cap 120k/20k
Kafka      max.poll.records=100                       # ALSO a liveness budget
           max.poll.interval.ms=300000                # batch must finish inside this

Kafka's equivalents are max.poll.records and fetch.max.bytes, and as the previous section showed they are not merely memory knobs — they are coupled directly to the liveness deadline. Prefetch set too aggressively in RabbitMQ costs you dispatch fairness. A batch set too large in Kafka costs you the partition.

Ordering guarantees cost you parallelism

Order is the property people request casually and pay for structurally. A best-effort queue lets any worker take any message, so parallelism is bounded only by how many workers you are willing to run. Every ordering guarantee removes some of that freedom, and the bill always arrives in the same currency.

SQS FIFO scopes order to a message group and permits one batch in flight per group at a time, so maximum parallelism equals the number of distinct groups, not the number of consumers deployed. Setting MessageGroupId to a customer identifier across a million customers is fine; setting it to the constant "orders" because it looked tidy gives you a single-threaded pipeline that no amount of autoscaling will accelerate, and the symptom is a queue that will not drain while every consumer sits idle. Kafka scopes order to a partition and caps useful consumers per group at the partition count — the thirteenth consumer on a twelve-partition topic is assigned nothing and simply burns money — and raising the partition count later changes which partition a key hashes to, so the guarantee has a discontinuity precisely at the moment you scale.

RabbitMQ scopes order to a queue, and there is a subtlety here that costs real incidents: the broker guarantees the order in which it dispatches, not the order in which work completes. Two consumers on one queue receive messages in order and then run concurrently, so message 2 routinely finishes before message 1. Broker-level FIFO gives you an ordered handout; only a single consumer at prefetch 1, or an application-level assignment of keys to dedicated lanes, gives you ordered effects. If the requirement is "all events for account 88 applied in sequence", the guarantee you actually need is per-key single-threaded consumption, and every broker makes you build it the same way — hash the key to a lane, and never process two messages from one lane at once. The semantics of guarantee scope, and what happens to ordered lanes when a message fails, are developed further in pub/sub system design.

Priority and delayed delivery

Priority is the feature most often requested and most often regretted. Kafka has none by design: a log has exactly one order and the entire storage model depends on it. SQS has none. RabbitMQ offers x-max-priority, and it applies only within the set of messages currently ready on the broker — anything already sitting in a consumer's prefetch buffer will not be reordered by a later high-priority arrival, so a generous prefetch silently defeats priority altogether. Under light load priority does nothing at all, because everything runs immediately; under sustained heavy load, strict priority starves the low tier indefinitely.

The production answer is nearly always separate queues per class with a deliberate allocation of workers, because that turns an implicit scheduling policy into an explicit capacity decision you can reason about, alert on, and change without a broker restart. If you genuinely need priority semantics with fairness, aging and starvation control, treat it as a service design in its own right — see priority queue service architecture.

Delayed delivery is better supported and still has sharp edges. SQS offers DelaySeconds up to fifteen minutes, set per message or per queue. RabbitMQ's common trick is a holding queue with a message TTL that dead-letters into the real queue, and that trick carries a failure mode that surprises everyone who builds it: TTL expiry is evaluated at the head of the queue only. A message with a ten-second TTL sitting behind one with a one-hour TTL is not released for an hour. The delayed-message exchange plugin exists because of exactly that. Kafka has no delay primitive; the workarounds are a fixed ladder of delay topics or an external scheduler that re-produces the record when it comes due.

The general rule is that brokers schedule badly. If you need arbitrary per-item due times, retries measured in days, or the ability to cancel work that has already been scheduled, what you want is a job store keyed by due time with the message as a pointer — something in the shape of Cloud Tasks — not a queue holding the work hostage in a buffer you cannot query.

Log-based brokers versus queue brokers

Strip away the marketing and the difference reduces to one question: where does per-message state live?

A queue broker — RabbitMQ, SQS, ActiveMQ — owns it. The broker knows which messages are ready, which are leased and to whom, how many times each has been delivered, and when each lease expires. Consumption is destructive: acknowledgement deletes. That bookkeeping is exactly what buys per-message operations — redeliver this one, delay this one, move this one aside after five attempts, delete this one — and it is exactly what it costs, because the broker maintains a mutable index over individual messages and becomes a coordination point as volume climbs.

A log broker — Kafka, Kinesis, Pulsar in its log mode — owns none of it. A partition is an append-only file. A record has an offset and nothing else. Reading mutates nothing, and records leave only when retention expires. The consumer stores its own position. That is why the throughput is different in kind rather than in degree: sequential writes, batched and compressed, streamed to consumers with almost no per-message server-side work. It is equally why per-message operations do not exist. You cannot delete one record, delay one record, or redeliver one record without redelivering everything after it.

Backlog visibility is not symmetric

A queue broker knows its own depth because it holds the state, so depth and oldest-message age are direct readings you can trust. A log broker has to derive the same information, and the derivation can mislead. Lag is the log-end offset minus the committed offset, per partition, and it goes wrong in two familiar ways. A consumer that commits before it processes reports zero lag while quietly dropping work, so the dashboard is green during data loss. And lag counted in records says nothing about time to drain unless you convert it using current throughput, which is why time-based lag is the number worth paging on. Lag is also per partition, so a healthy aggregate can hide one partition stuck for an hour behind a poison record: the fleet looks fine, and one customer's data has silently stopped moving.

Failure handling is native in one and manual in the other

Queue brokers ship the retry machinery: a delivery counter on each message, a configured maximum, and an automatic move to a dead-letter destination when it is exceeded. A log broker has no counter and nowhere to move a record to, so the consumer owns the whole policy — catch the exception, decide whether it is retryable, produce a copy to a separate error topic, commit the offset past the bad record, and reconcile later. Both models eventually face the same questions about retry budgets and quarantine, which are worked through in dead-letter queue architecture; what differs is who writes the code. Stronger delivery guarantees split the same way. Kafka offers a transactional produce-and-commit inside its own boundary (exactly-once semantics), queue brokers offer at-least-once and expect an idempotent consumer, and neither helps across the boundary into a database — which is the gap the transactional outbox exists to close.

Which properties actually force the choice

Most comparisons enumerate features, which is why they rarely settle an argument. Only a handful of properties are genuinely non-negotiable, and the decision should turn on those alone.

A log broker is forced when the same data has multiple independent consumers you cannot enumerate in advance, and adding the fourth one must not require touching the producer. When replay is a requirement rather than a comfort: a new service needs the last thirty days on first boot, or a bug means state has to be re-derived from history. When volumes reach hundreds of thousands of records per second, where per-message broker bookkeeping is simply not affordable. And when the data is a stream of facts from which several different views are derived — the shape that event sourcing and stream processing assume from the start. Kinesis occupies the same architectural slot as Kafka with the operations outsourced.

A queue broker is forced when messages are work items with variable and sometimes long durations, so one slow item must not stall a lane. When per-message retry, delay and dead-lettering are hard requirements you would rather configure than implement and then maintain. When worker count must far exceed any sane partition count — eight hundred workers chewing through thirty-second jobs is unremarkable for a queue and absurd as a partition count. When you need to reach in and act on one specific message. And when the operational budget is effectively zero, which is the honest reason most teams should choose SQS and stop deliberating.

Properties that do not force the choice, despite being cited in every meeting: durability, since both replicate before acknowledging a write; ordering, since both offer per-key ordering with a documented scope; and possible future fan-out, since SNS-to-SQS and RabbitMQ exchanges both fan out perfectly well. Cost usually favours the queue at low and moderate volume, because a managed queue is billed per request with ten-message batching dividing that by ten, while a Kafka cluster is billed as always-on brokers and storage whether or not anything is flowing through it.

The two models also compose, and large systems generally end up composed rather than pure. The log becomes the system of record and the integration bus; a small bridge consumer reads it and enqueues discrete work items onto a queue per work type, where retries, delays and dead-lettering are cheap and per-message. You get replay and open-ended fan-out at the front, and per-message operational control at the back, at the price of one more hop and one more idempotency boundary to get right.