API Gateway Patterns

Cross-cutting concerns at the gateway

Authentication and authorization is the first concern. Rather than embedding JWT validation logic in every backend, the gateway validates the token, extracts claims (user ID, scopes), and passes them to backends via headers or request context. If the token is invalid or expired, the gateway returns 401 immediately, never touching the backend. This scales: one gateway validates a thousand requests; a hundred backends don't repeat the work.

Rate limiting is the second. Without it, a single client hammering an endpoint starves other callers. The gateway tracks requests per API key, IP, or user ID against a quota (e.g., 1000 requests per hour per key). Once a client hits the limit, the gateway returns 429 immediately, shedding load before it reaches the backend. This is cheap: a counter and a timestamp in memory, checked before forwarding.

CORS headers, header injection, and request logging are also handled here. The gateway adds Access-Control-Allow-* headers for cross-origin requests, injects trace IDs for observability, removes sensitive headers before forwarding, and logs every request. Backends see only the headers the gateway allows and the trace context the gateway provides, making security and debugging centralized and auditable.

Routing strategies — the gateway's core job

The gateway routes every request to the right backend. Path-based routing is the simplest: if the path starts with /users, route to the users service; /orders goes to the orders service. The gateway maintains a list of rules, each with a path pattern and an upstream address (a service name in Kubernetes, a URL in a cloud deployment). No backend knows about the others; the gateway is the only place that knows the full topology.

Host-based routing is equally simple: route api.example.com to one backend and admin.example.com to another, all handled by one gateway. This is cheaper than running a separate reverse proxy for each hostname and cleaner than having each backend handle multiple domains.

Weighted routing enables canary deployments. Route 95% of traffic to the stable version and 5% to the new version. If the new version's error rate stays low and latency is acceptable, shift more weight to it gradually over hours. If it breaks, roll back by shifting weight back to 100% stable. This happens at the gateway, not by changing code or configs in the backends themselves — the gateway is a single point of control for the entire fleet.

Rate limiting and quota enforcement

Rate limiting is not just about hammering a number back at the client. The gateway must pick a key: per API key (each registered client gets a limit), per IP address (every IP gets a fair share), or per user ID (extracted from the JWT and matched to a tier). Then measure: count requests in a rolling window (last 60 seconds) or fixed windows (per minute, per hour). When the count exceeds the quota, return 429 Too Many Requests immediately.

The tricky part is distributed rate limiting. If the gateway runs on five servers, each one tracking rates in its own memory, they disagree: each thinks the client has made 200 requests when they've made 1000 total across the fleet. The fix is to push rate state to a fast, central store (Redis, Memcached) so all gateways see the same counters. The latency cost is one Redis round-trip per request, but Redis is optimized for this and is cheap compared to letting a malicious client take down a backend.

Also consider quota refund: if a client's request lands in the gateway but the backend returns 5xx (the backend's fault, not the client's), should the rate limit count against the client? Most systems refund to be fair: the gateway only counts successful requests or explicit client errors (4xx), not infrastructure failures.

Request and response transformation

An older client speaks API v1 (returns user objects with a fullName field). The backend was updated to v2 (now only returns firstName and lastName separately). The gateway bridges the gap. On the response, it sees v2 data, and if the request came from a v1 client, it transforms the response back to v1 (concatenate firstName + ' ' + lastName to synthesize fullName). The client is happy; the backend does not maintain multiple schema versions.

Similarly, the gateway can add or remove fields on the way in. A legacy client sends user_id (snake_case); the backend expects userId (camelCase). The gateway rewrites the request. Or a client includes deprecated fields that the backend now rejects; the gateway strips them before forwarding, and the request succeeds. This is centralized versioning: the backend is always the latest version, and the gateway handles all the backward compatibility.

API versioning without duplicating backends

A common pattern is to version the API in the URL path: /v1/users and /v2/users both exist, but internally route to the same backend. The gateway applies transformations based on the version flag. This way clients can upgrade their version string in their URL whenever they are ready, and the backend never splits into two codebases.

Another pattern is to route different versions to different backends during a migration. Route /v2/users to a new, rewritten backend (maybe in a different language or architecture) while keeping /v1/users on the old one. Gradually migrate clients to v2, then shut down v1. The gateway is the switchboard: clients see a stable URL (the gateway), but the underlying backend can be changed, split, or merged without the client knowing.

Error handling and fallback strategies

When a backend is slow or down, the gateway has options beyond returning 502 Bad Gateway. First is a timeout: wait for the backend for 5 seconds; if no response, fail fast instead of holding the connection open for 30 seconds and tying up a thread. Second is a circuit breaker: if the backend fails repeatedly, stop sending traffic to it for 30 seconds (the cooldown), giving it time to recover. After cooldown, try one probe request; if it works, resume normal traffic. If it fails, open again.

Fallback responses soften the impact. If the recommendation service is down, return an empty list of recommendations instead of 500 — the main user data still loads, the page is still usable. Pair this with cache: if the backend is down but the gateway cached the last successful response (expires in 5 minutes), serve stale data. The user sees a minute-old result instead of an error. For APIs, this is a graceful degradation pattern: the system continues in a reduced capacity instead of failing hard.

Deployment patterns — sidecar vs. edge vs. central

Sidecar deployment runs a lightweight gateway instance next to each backend service (common in Kubernetes service meshes like Istio). The sidecar sees all traffic to that backend and handles auth, rate limiting, and routing. The benefit: each service can have its own gateway rules, and the gateway sees the full request context. The downside: you maintain N gateways (one per service), and cross-service concerns require distributed coordination.

Central gateway deployment runs one or a few gateway instances that all services route through. All auth, rate limiting, and routing rules live in one place. The benefit: central control, easy to update a rule affecting all backends. The downside: single point of configuration (though not failure, if replicated), and the gateway becomes a scaling bottleneck if it cannot keep up with traffic.

Edge deployment puts the gateway at the edge (CDN nodes, regional POPs) close to clients. Authentication and early routing happen there, then requests are routed to regional backends. Reduces latency and shields origin infrastructure. Common in large-scale, geographically distributed systems.

Monitoring, observability, and debugging

The gateway is the lens through which every request passes. Metrics flow from the gateway: request count, latency (p50, p99), error rate, rate-limit rejection rate. Alert when the gateway's error rate sikes or latency jumps; it signals trouble in the system. Also track backend health: which backends are erroring? Which are slow? The gateway sees this first and can make routing decisions (shed load from a slow backend, circuit-break a dead one) before the load trickles back to you via application alerts.

Tracing ties it together. Generate or forward a trace ID to every request. The gateway logs the trace ID in every decision it makes (auth result, rate-limit check, backend choice). When a user reports a problem, look up their request by ID and replay the gateway's logic: was the request rejected? Slow? Routed to the wrong place? The trace provides a full story. In a distributed system, this is critical for debugging.

Access logs are also invaluable. Log every request (client IP, method, path, authenticated user, backend chosen, response latency, status code) to a searchable store (e.g., ELK, Splunk). Run queries like 'requests to backend X in the last hour' or 'latencies for user Y' and debug production issues quickly.

Platforms compared — Kong, Ambassador, AWS API Gateway

Kong is an open-source, on-premises gateway based on Nginx. Highly extensible via Lua plugins (write custom auth logic, transformations, anything) and has a rich plugin ecosystem. You run it yourself on VMs or Kubernetes, manage its database (PostgreSQL or in-memory), and pay nothing for the software. Trade-off: operational burden. You tune Nginx config, scale instances, backup the database. Best for teams with ops expertise and custom requirements (complex rate limiting logic, proprietary auth schemes).

Ambassador (now part of the Emissary ingress family) is built on Envoy and is Kubernetes-native. Declare your routes in Kubernetes CustomResources; no separate config database. Integrates tightly with Kubernetes service discovery so you don't manually list backends. Also open source with a commercial support option. Scales horizontally within Kubernetes. Best for Kubernetes shops that want an ingress/gateway without external databases.

AWS API Gateway is a managed service: no infrastructure to manage, scales automatically, integrates with IAM for auth, and charges per request (generally $0.35 per million requests). Set up routes in the AWS console or via CloudFormation, and it just works. Downside: less customizable than open-source platforms, vendor lock-in. Best for teams on AWS that want to minimize operational overhead and can tolerate slower feature iteration.

Integration with microservice architectures

The gateway is the entry point to the microservice fleet but is not the only place where routing and load balancing happen. Service meshes (Istio, Linkerd) run sidecars next to each service for inter-service communication. The gateway handles north-south traffic (client to backend); the service mesh handles east-west traffic (service to service). Both maintain circuit breakers, rate limiting, and tracing, but at different layers.

In practice, the gateway and service mesh must coordinate. A circuit breaker at the gateway prevents hammering a dead backend; a circuit breaker in the service mesh prevents a downstream service from hammering an even further downstream one. Use the gateway for coarse, client-facing concerns (client rate limits, public API versioning, authentication). Use the service mesh for fine-grained, service-to-service concerns (latency budgets, retries with exponential backoff, timeout cascading). This separation of concerns keeps both layers lean and debuggable.

Best practices and design principles

Keep the gateway stateless. Deploy multiple gateway instances and run a load balancer in front of them. If one fails, traffic shifts to another. Session state (rate-limit counters, circuit breaker history) must live in a central cache (Redis) so all instances agree.

Tune timeouts carefully. A gateway timeout that is too long holds resources waiting for a slow backend, which starves other requests. Too short and you give up on requests that would have succeeded if you waited. Measure your backends' latency and set the timeout to 2-3x the 95th percentile, not the worst-case (which could be infinite).

Monitor the gateway itself. It is easier to notice when a backend breaks (users complain) than when the gateway becomes a bottleneck (throughput stays the same but requests queue longer). Alert on gateway latency, CPU, and memory so you scale before it breaks. Also alert on configuration churn: a gateway that reloads rules a hundred times a day is flaky; a stable rule set (reloaded rarely) is a sign of a well-tuned system.

Version your gateway rules. If a rate-limit change breaks a key client, you need to roll back in seconds. Store rules in version control, deploy via CI/CD, and keep a history so you can diff what changed between deployments. This prevents 'mysterious gateway behavior' issues and makes troubleshooting reproducible.