Load Balancing — L4 vs L7, Algorithms, and Health Checks

Why it matters

Every production system needs load balancing. Getting it right is invisible; getting it wrong causes cascading failures. Understanding the choices lets you architect systems that scale predictably.

The architecture

L4 (transport) load balancing operates on TCP/UDP. Fast, protocol-agnostic, but can't inspect payload. Examples: AWS NLB, HAProxy in TCP mode.

L7 (application) load balancing operates on HTTP. Can route by path, header, cookie. Slower but far more flexible. Examples: AWS ALB, NGINX, Envoy.

How it works end to end

The sections below work through this in the order the decisions are actually made: what each layer is able to see and what that costs, how one healthy backend gets chosen over another and what each policy quietly assumes, how members join and leave without dropping traffic, and the failure modes that turn a balancer from a shock absorber into an amplifier.

What an L4 balancer can see, and what an L7 proxy buys

A transport-layer balancer makes exactly one decision per flow, and it makes it from the five-tuple: source address, source port, destination address, destination port, protocol. That is the entire input. It cannot tell whether the flow will carry a single request or two hundred thousand, whether the bytes are HTTP or the Postgres wire protocol, or whether the backend answered 200 or 503. Forwarding is either address translation, which keeps per-flow state in a connection table, or direct server return, where the backend replies to the client directly and the balancer never observes the response path at all. Direct return is dramatically cheaper — the balancer only touches the request half of the traffic — and it is also the moment the balancer becomes structurally blind to how requests are going.

Because it never parses the payload it also never terminates TLS, and that is occasionally the point: encryption stays end to end, no private key sits on the proxy fleet, and there is no decryption point to argue about in a compliance review. The price is that everything the ciphertext conceals is unavailable to you. There is no routing by path or header, no injected request identifier for tracing, and the backend sees the balancer as its peer unless you add the PROXY protocol preamble to recover the original client address. SNI-based steering claws back the requested hostname and nothing else, because SNI is the one field sent in the clear.

An application-layer proxy terminates the connection, parses each message, and decides per request. That is what pays for host and path routing, header rewriting, buffered retries, per-route timeouts, request-level telemetry, and affinity keyed on a cookie rather than on a client address. It costs CPU — a handshake plus a parse rather than a table lookup — and it changes the failure model, because the proxy now holds buffers and upstream connections that a stateless flow forwarder never had.

Affinity is the sharpest illustration of the gap. Source-address stickiness is the only kind a transport-layer device can offer, and it is least reliable for the clients you have most of: mobile users change address mid-session, and carrier-grade NAT hides tens of thousands of subscribers behind a handful of addresses, so one hash bucket can drop an entire access network onto one backend. Cookie affinity is per user, survives address changes, and can be given an explicit lifetime — but it needs someone to read the request.

Round robin and least-connections: what each one assumes

Rotation is correct under two conditions that production rarely satisfies: every backend has the same service rate, and every request costs roughly the same. Violate the first — a fleet that has drifted across three instance generations, or where a quarter of the machines share a noisy neighbour — and equal request counts produce unequal utilization. Nothing alerts, because the balancer is doing precisely what it was asked. The slow machines just carry deeper queues, and the symptom surfaces as a tail-latency mystery rather than as a distribution problem. Violate the second — a workload mixing 2 ms cache reads with 800 ms report builds — and rotation distributes requests evenly while distributing work at random.

The third failure is structural rather than statistical. Rotation balances whatever it counts, and if the unit it counts is connections rather than requests, a protocol with long-lived connections gets balanced once, at connect time, and never again. Ten thousand WebSocket sessions spread evenly across ten machines is a fine picture until you scale to twenty: the ten new machines take every new session and none of the established ones, leaving half the fleet loaded and half idle, with no way to level it that does not disconnect users.

Least-connections swaps the counter for outstanding requests, which is a far better load proxy because it is closed-loop. A backend that slows down accumulates in-flight requests and is avoided automatically, without anyone measuring or publishing latency. Its own hidden assumption is that a pending request means comparable work everywhere. Where it fails is the empty queue: a backend with nothing outstanding is the most attractive target this policy can imagine, regardless of why it is empty, and the two most common reasons are that it started thirty seconds ago and that it is rejecting every request in under a millisecond. Both get rewarded with the full arrival rate. Fast failure is indistinguishable from idle capacity if the only thing you count is depth.

Power of two random choices

This is the idea worth internalising, because it explains why serious balancers stopped trying to be clever about global state.

Start with the baseline. Throw n requests at n backends uniformly at random. The average load is one, but the maximum is not: the busiest backend ends up holding on the order of log n / log log n of them. The important property of that expression is that it has no ceiling — the imbalance produced by pure chance keeps growing as the fleet grows, so you cannot outrun it by adding machines. The obvious repair is to always send to the least-loaded backend, which requires knowing the load of every backend at the instant of every decision. Inside one process that is a scan on the hot path. Across fifty balancer processes it is a distributed agreement problem, and the scoreboard you build to solve it becomes both a bottleneck and a new thing that can fail.

The result that changes the economics: sample two backends uniformly at random and send the request to whichever of the two is less loaded. The maximum load drops to the order of log log n / log 2 plus a constant. Doubly logarithmic is the whole point — that quantity grows so slowly that it is effectively a small constant across every fleet size you will ever operate, and going from a hundred backends to a hundred thousand barely moves it. You bought that with two lookups instead of n. Sampling d candidates rather than two merely replaces log 2 with log d in the denominator, so the third sample buys a slim margin and the tenth buys nothing you can measure. Almost the entire benefit arrives with the second sample, and that is the whole trick.

Two consequences matter in operation. The first is that the decision is local. A balancer needs a load figure for two candidates, not for the fleet, so a hundred independent proxies can each run this from private counters and the aggregate distribution still comes out well. The second is the one that saves you: independent deciders that each pick the globally least-loaded backend all pick the same one, converge on it, bury it, then flee together to the next victim. That oscillation is the characteristic signature of greedy least-loaded across multiple deciders, and random sampling desynchronises them at no cost.

The real caveat is staleness. The analysis assumes the load reading reflects the present. When the reading lags, taking the minimum of two amplifies whatever error the lag introduced, and with sufficiently old information the sampled minimum can end up worse than choosing at random, because every decider races toward the same obsolete bargain. That is why implementations count the requests the proxy itself has outstanding to each candidate — a number that is exact, free, and current by construction — instead of consuming a published utilization metric that is several seconds behind. Latency-aware variants keep that property while fixing its blind spot: weight each candidate by a decaying estimate of its recent response time multiplied by its in-flight count, so a backend that is simply slower per unit of queue is penalised correctly. Envoy's least-request policy and Linkerd's peak-EWMA balancer are both this shape.

choose(pool):
    a = pool[rand()]                 # two uniform samples, never a scan
    b = pool[rand()]
    # cost() is local and current: requests this proxy has outstanding,
    # optionally x decayed recent latency, divided by the backend weight
    return a if cost(a) <= cost(b) else b

One deployment detail undoes all of it: if the candidate pool is a subset of two or three backends — because a client was given a tiny slice of the fleet — then sampling two of them is sampling nearly all of them, and you are back to greedy least-loaded with its herding behaviour. Sampling only helps when there is a meaningful population to sample from.

Affinity, consistent hashing, and bounded load

Sometimes the routing decision is not free to be random, because the backend holds something the request needs: a warm local cache, an in-memory session, an open upstream connection, a partition of state. Hashing a stable request attribute — user id, tenant, cache key — onto a ring gives you that affinity while keeping reassignment proportional to the membership change rather than total. The mechanism, the ring, virtual nodes, rendezvous and jump variants, and the capacity-capped forwarding rule are covered in depth in consistent hashing architecture, and the short version of why you need the capped variant is worth repeating here: hashing is uniform in expectation and lumpy in the moment, and request popularity is never uniform, so whichever backend owns the arc containing your hottest key runs hot while its neighbours idle.

The tension with everything above is direct. Every unit of affinity you add is a unit of balancing freedom you give up, and the two policies pull against each other under exactly the conditions where you need both. Treat affinity as an optimisation with a fallback: if the hash target is unhealthy or over its cap, the request must still be servable elsewhere, more slowly. An affinity scheme that produces errors rather than cache misses when its target disappears has quietly turned a stateless tier into a stateful one. Hot keys that survive the capping rule are a different problem with different tools — see hot-key mitigation.

Weights, slow start, and the cold-backend trap

Weighting is the honest answer to a heterogeneous fleet: divide each backend's measured cost by a capacity factor so a machine with twice the cores attracts twice the traffic. Weights compose cleanly with the sampling policy above, since they only change how you compare two candidates. They do not compose cleanly with human maintenance, because weights are static configuration describing a fleet that changes underneath them, and a stale weight is worse than no weight at all.

The interesting case is the backend that is new. A freshly started process has a cold page cache, an empty connection pool to its own dependencies, an unwarmed JIT, and lazily initialised singletons that have not been touched yet. Its steady-state capacity might be identical to its peers and its capacity in the first thirty seconds a fraction of that. Now combine that with least-connections or with sampled least-loaded: the new backend has zero outstanding requests, so it looks like the most attractive destination in the fleet, and it receives a disproportionate share of arrivals at precisely the moment it can handle the least. It slows, its queue grows, and if a health check with a tight timeout is watching, it gets ejected for being unhealthy — which is why an autoscaling event during a traffic spike can produce a stream of instances that never survive their first minute.

Slow start is the fix, and it is a ramp rather than a switch: for a configured warm-up window, scale the new backend's effective weight from near zero up to its full value, usually linearly against elapsed time since it first passed a check. The window has to be set from the thing that is actually cold. A JVM service with a large working set and a fifteen-second warm-up window will still be knocked over; measure the time it takes for the instance's latency curve to converge with its peers, and set the ramp longer than that. The same ramp belongs on the return path for a backend that was ejected and has recovered, for identical reasons.

Health checks that do not remove the whole fleet

An active check is a probe on a timer, and its parameters set your detection latency arithmetically. With a three-second interval, a one-second timeout, and a threshold of three consecutive failures, a backend that dies immediately after a successful probe is still in rotation for up to about eleven seconds. Tighten the numbers and detection speeds up while your false-positive rate climbs, because a single garbage collection pause or a dropped packet now counts as evidence. This is a genuine trade with no correct answer, only an answer calibrated to how bad a wrongly ejected backend is for your particular fleet size.

The dangerous mistake is the check that is too deep. A readiness endpoint that verifies its database, its cache, and three downstream services before answering seems thorough, and it is — it thoroughly converts any shared-dependency blip into simultaneous ejection of every instance, because they all fail the same check at the same moment for the same reason. Health signals must be independent across backends to be useful; a check that reads a shared dependency is perfectly correlated by construction. The rule that survives contact with production is that a readiness probe answers "can this process serve a request", not "is the wider system well". If a dependency is down, degrade or fail the request and let that show up in error rate and in outlier detection, where the response is proportional. The liveness, readiness, and startup distinction is worked through separately in health check architecture.

Because correlated failure is possible anyway, a balancer needs a floor. The standard mechanism is a panic threshold: when the healthy fraction of a pool drops below some percentage — fifty percent is the common default — the balancer stops honouring health status and distributes across every member, healthy or not. That sounds reckless and is not. Once most of the fleet has been marked bad, the marks are far more likely to be wrong, or the survivors are about to be crushed by traffic redistributed from everyone else. Spreading across a possibly-sick fleet beats concentrating all load onto the two instances that happened to answer their last probe.

The other half of the answer is to judge a backend by its actual output. Outlier ejection derives its verdict from the requests you are already sending rather than from a synthetic probe, removing a backend whose consecutive server errors cross a limit, or whose success rate falls a configured number of standard deviations below the fleet's. It catches partial failures that probes miss — one bad shard, one corrupt cache, errors that only appear on real request shapes — and it reacts in seconds because the evidence is already flowing. It needs the same floor: cap the fraction of the pool that may be ejected passively, and make each re-ejection last longer than the previous one so a flapping instance is quarantined for progressively longer instead of being retried every ten seconds. Note what neither mechanism detects well: overload. A saturated backend is often the last one to fail a cheap probe, because answering a trivial endpoint stays fast long after real requests have stopped meeting their deadlines. Shedding, not ejection, is the tool there — see load shedding.

Draining and graceful shutdown

Removing a backend cleanly means it stops being chosen for new work and is then allowed to finish what it already accepted. Almost every implementation gets the first half right and the ordering wrong. In a typical orchestrated rollout, the termination signal reaches the process and the endpoint removal is published to the balancers at the same moment, and those are not the same moment at all: the signal is delivered in microseconds, while the removal must propagate through a registry, be observed by every proxy, and take effect. For the seconds in between, a process that has already begun shutting down is still being sent requests, and every one of them becomes a connection reset that a user sees. The fix is unglamorous — a pre-stop delay that does nothing but sleep, long enough to cover propagation, before the application starts tearing anything down. Deregistration delay on a managed balancer is the same idea from the other end.

The signalling mechanics differ by protocol and are worth knowing. On HTTP/1.1, a server can set Connection: close on responses so keep-alive connections retire naturally as they are used. On HTTP/2 the equivalent is GOAWAY, and the graceful form is two-phase: send an initial GOAWAY carrying the maximum possible stream identifier, which announces intent without invalidating any stream in flight, wait a round trip so the peer stops opening new ones, then send a second GOAWAY naming the last stream you will actually process. Skipping the first phase races the client and kills requests it had already dispatched. At the transport layer there is no drain signal at all — a flow forwarder can only stop selecting a backend for new flows and wait for the existing ones to end, which for a persistent connection may be never, hence the hard drain timeout. Set that timeout above your longest legitimate request, and remember that a streaming endpoint or a long poll makes "longest legitimate request" a design decision rather than a measurement.

Connection-level balancing versus request-level

Everything a transport-layer balancer does is decided once per connection, which was fine when connections were plentiful and short. It stops being fine the moment your protocol multiplexes many logical operations over one long-lived connection: HTTP/2 and gRPC by design, but also database drivers holding pooled sessions, message-broker clients, MQTT, and anything built on WebSockets. A handful of clients open a handful of connections, those connections are pinned at setup, and the load distribution you observe is the distribution of connections, not of work. Scaling the backend fleet moves nothing, because no new connections are being made. The gRPC-specific mechanics and the standard remedies are covered in gRPC architecture.

The general shape of the fix is one of three moves. Put a proxy in the path that understands the framing and dispatches each stream independently, which restores per-request balancing at the cost of a hop and a parse. Move the decision into the client, which is where the freshest information already lives. Or force churn: cap the maximum age of a connection so clients periodically reconnect and re-resolve, which spreads them over new members without anyone noticing. That last one is a blunt instrument and needs jitter, or every client that connected during the same deploy will reconnect during the same second.

Client-side and lookaside balancing

A client that balances for itself has the best possible information: it knows exactly how many requests it has outstanding to each backend, with zero staleness, and it pays no extra network hop. It also needs the entire policy — sampling, weights, ejection, draining, retry budgets — implemented and kept consistent in every language your organisation deploys, which is the reason this approach concentrates in monoglot shops and in ecosystems with one dominant RPC library. Lookaside balancing splits the difference: a control service watches membership and load and tells each client which backends to use, while the data path stays direct from client to backend. The heavy thinking is centralised, the hop is not.

Both hit the same wall at scale, and it is a connection wall rather than a CPU one. If every one of C clients connects to every one of S backends, the fleet holds C times S connections, and at a few thousand of each that is tens of millions of sockets, keepalives, and health probes that nobody budgeted for. Subsetting is the answer: each client is assigned a deterministic slice of the fleet, sized by its offered load, so connection count grows with the subset size rather than with the fleet. Choosing the slice deterministically matters, because a randomly chosen subset per client leaves some backends serving four clients and others serving none. Size the subset with the sampling caveat above in mind — a slice of three defeats the point of random sampling entirely, while a few dozen backends per client keeps both properties. The sidecar variant, where the policy lives in a proxy next to each workload rather than inside it, is covered in service mesh architecture; membership itself is service discovery.

The tier above the balancer: DNS, anycast, and global steering

Local balancing chooses an instance. Something above it chooses a region, and that layer runs on different physics: decisions are coarse, slow to propagate, and made by resolvers and routers you do not control. Name-based steering is the common approach and its weakness is cached answers — record lifetimes are advisory, intermediate resolvers round them up, and some client runtimes cache resolved addresses for the life of the process, so a record you changed thirty seconds ago is still steering real traffic to a region you withdrew. Anycast sidesteps caching by moving the decision into routing itself, at the cost of a different hazard: when routes reconverge, packets belonging to an established connection can arrive at a different site that has no state for them, which is survivable for stateless request-response and not for a long transfer. Both mechanisms are worked through in DNS failover and traffic steering and anycast architecture. Keep the two layers apart for a mechanical reason rather than an aesthetic one: a global decision can only be as timely as the propagation delay of the signal it acts on, so a steering layer that reacts faster than its own health data converges will flap regions in and out on noise. The local layer has no such limit, because its evidence is the traffic passing through it right now. Fuse them and you get one component governed by the slower of the two constraints.

Failure modes worth designing against

Retry amplification is the most common way a balancer turns a bad minute into an outage. Retries are load, and they arrive exactly when the system has the least capacity to absorb them. Worse, they multiply through tiers: if each of three layers retries twice on failure, one user request can become up to twenty-seven backend requests during a partial failure, which guarantees that a system trying to recover cannot. Two mechanisms hold this down. A retry budget expresses retries as a fraction of successful traffic — commonly ten to twenty percent — so retrying stops being possible as soon as most requests are failing, making storms arithmetically impossible rather than merely discouraged. And a deadline propagated with the request lets each layer refuse to issue a retry it cannot possibly complete in the time remaining, which removes the most wasteful attempts first.

Failover produces a herd. When a backend, a zone, or a region is withdrawn, every client that was using it reconnects at the same instant, and reconnection is far more expensive than steady-state traffic: a fresh TLS handshake costs orders of magnitude more CPU than a request on a warm connection, and the surviving capacity now faces a synchronised handshake burst on top of the traffic it inherited. Systems that survive this jitter everything that could synchronise — backoff intervals, connection lifetimes, health probe schedules, cache expiry — and ramp recovered members in gradually rather than restoring them to full weight the moment they pass a probe.

Then there is the balancer itself, which you introduced to remove a single point of failure and which is now one. The standard answer is to run many balancer instances behind a shared address advertised from all of them, with equal-cost routing spreading flows across the set (see ECMP). That works, with a sharp edge: adding or removing a balancer node rehashes flow-to-node assignments, and every connection whose assignment moved lands on a node with no state for it and dies. Balancers that survive their own scaling events use a hashing scheme whose assignments barely move on membership change, or share flow state between nodes, or both. Capacity limits are physical too — connection tracking table entries, ephemeral port exhaustion when translating many flows toward few backends, handshakes per second — and none of them appear in a request-per-second estimate. The most frequent real cause of a total balancer outage, though, is none of these: it is a configuration change, which by design reaches every proxy in the fleet within seconds and therefore breaks all of them simultaneously. Route and policy configuration deserves the same staged rollout, canary, and fast revert as application code, because it is deployed more often and validated less.