Long polling — HTTP server push, cursors, held connections

Why architecture matters here

Long polling matters because compatibility is sometimes worth more than efficiency. WebSockets are more efficient for real-time bidirectional communication, but they require a protocol upgrade that some network intermediaries block, they need fallback handling, and they add complexity. For a use case where updates are occasional (not high-frequency), where the audience's networks are unpredictable (corporate proxies, restrictive firewalls), or where simplicity matters, long polling's 'works everywhere HTTP works' is a decisive advantage. Notification delivery, status updates, chat with modest message rates, and any server-push need where reaching every client reliably beats squeezing out efficiency — these are long polling's domain, and choosing it deliberately (rather than reaching for WebSockets reflexively) is sometimes the right architectural call.

The reliability challenge long polling must solve is the gap between responses. Between the server responding (with data) and the client's next request arriving, there's a window where new data could arrive with no held connection to deliver it on — and a naive implementation loses those messages. The solution is a cursor (or since-token): the client tracks what it's received (a message ID, timestamp, or sequence number), sends it with each request, and the server returns everything since that cursor — so messages arriving during the gap are delivered on the next request rather than lost. This cursor discipline is what makes long polling reliable; without it, the gap between requests is a message-loss window, and the reliability difference between a correct long-polling implementation and a naive one is entirely in the cursor.

And the scaling reality is that held connections are the cost. Each long-polling client holds a connection open (waiting for data), so a server with many clients holds many connections — and if the server uses thread-per-connection (blocking IO), it exhausts threads quickly (each held connection parks a thread doing nothing but waiting). The answer is async servers (non-blocking IO, event loops, or virtual threads) that hold many connections cheaply — the same scaling requirement as WebSockets and SSE, because they all hold connections. A long-polling deployment that uses a blocking server hits a connection wall fast; one that uses async IO holds tens of thousands of waiting clients on modest resources. Understanding that held connections are the scaling constraint (and async servers the answer) is essential to deploying long polling at scale.

The architecture: every piece explained

Top row: the cycle. The client request asks for updates (typically with a cursor indicating what it's already seen). The server holds the request — instead of responding immediately, it waits (registering interest, parking the request asynchronously) until there's data for this client or a timeout is reached. Data or timeout: when data arrives (a new message, an update), the server responds with it; if no data arrives within the timeout (say 30 seconds), the server responds empty (a keepalive, preventing the connection from being killed by intermediaries and letting the client re-request). Client re-requests: on receiving a response (data or timeout), the client immediately makes a new request — so there's almost always a held request ready to receive the next push. The immediate re-request is what keeps the push channel continuous.

Middle row: reliability and tuning. Held connections are the resource: one per client, waiting — the scaling constraint, requiring async servers. Cursor / since-token is the reliability mechanism: the client tracks its position (last message ID/timestamp), sends it with each request, and the server returns everything since — so messages arriving in the gap between responses aren't lost; they're delivered on the next request. Timeout tuning balances latency and load: longer timeouts (fewer reconnections, less overhead, but connections held longer and intermediaries may kill idle ones); shorter timeouts (more reconnection overhead but more resilient to intermediary timeouts) — typically 20-60 seconds, tuned to the environment's intermediary timeouts. vs WebSocket/SSE: WebSocket for high-frequency bidirectional, SSE for server-to-client streaming, long polling for compatibility-critical or occasional-update cases — and long polling as the fallback when the others fail.

Bottom rows: scaling and role. Scaling held connections: async servers (non-blocking IO, event loops, or virtual threads) hold many waiting connections cheaply; the backend must support registering interest and notifying held requests when data arrives (pub/sub, a notification mechanism) rather than each held request polling the data source (which would be a database-hammering disaster). Compatibility is the key advantage: long polling works through every HTTP intermediary, in every client, with no upgrade — the reason it's the reliable fallback and the choice for restrictive environments. The ops strip: connection limits (held connections bounded by server capacity, sized for the client count), cursor durability (the cursor mechanism reliable across reconnections and server restarts, so no messages are lost), and fallback role (long polling as the graceful degradation when WebSocket/SSE are unavailable, with the client detecting failure and falling back).

End-to-end flow

Trace a notification system using long polling for compatibility. The application serves a corporate audience behind proxies that sometimes block WebSocket upgrades, so long polling is chosen for reliable reach. A client requests GET /notifications?since=1042 (cursor = last notification ID seen). The server checks: any notifications after 1042? No — so it holds the request (registering this client's interest with the notification pub/sub, parking the request asynchronously on the event-loop server). 20 seconds later, a notification (ID 1043) is created; the pub/sub notifies the held request; the server responds with notification 1043; the client displays it and immediately re-requests GET /notifications?since=1043. The push arrived in near-real-time (the moment the notification was created, the held request delivered it), over plain HTTP, through the corporate proxy that would have blocked a WebSocket.

The reliability vignette shows the cursor's importance. Between the server responding with 1043 and the client's re-request arriving, notification 1044 is created — a message in the gap. Because the client re-requests with since=1043, the server sees 1044 is after the cursor and returns it immediately (no holding needed — there's already data) — the gap message delivered on the next request, not lost. Had the implementation used no cursor (just 'give me new notifications'), 1044 would have been created with no held connection and lost. The cursor turned a message-loss window into reliable delivery — the difference between correct and naive long polling.

The scaling and fallback vignettes complete it. The system has 50,000 concurrent clients, each holding a connection — a blocking thread-per-connection server would need 50,000 threads (impossible); the async event-loop server holds all 50,000 cheaply (each is just a parked request awaiting a pub/sub notification, no thread). The backend uses pub/sub (not per-request database polling — 50,000 requests polling a database every second would melt it), so held requests are notified on data arrival efficiently. And long polling serves as a fallback in a different part of the product: a real-time feature primarily uses WebSockets, but when a client's WebSocket fails to connect (blocked by a proxy), it falls back to long polling — degraded efficiency but reliable delivery, the graceful degradation that keeps the feature working for everyone. The consolidated discipline: cursors for reliable gap-free delivery, async servers to hold many connections cheaply, pub/sub backends to notify held requests efficiently, timeout tuning for the environment's intermediaries, and long polling chosen deliberately for compatibility-critical cases or as the fallback for richer protocols — because it's less efficient than WebSockets but works everywhere HTTP does, and sometimes that reach is worth the overhead.

What is long polling? A short answer and a working example

Long polling is a way for a server to push data over ordinary HTTP. The client sends a request and the server does not answer until it has something to say or a timeout expires. As soon as a response arrives, the client sends the next request, so there is nearly always one request waiting at the server. It uses no protocol upgrade, which is why it passes through proxies that block WebSockets.

The program below is a complete long-polling server and client using only the Python standard library over a real localhost socket. Help-desk updates for a case at Belgavi's Book Store are published to a list, each with an increasing id, and the client passes the last id it saw as since. Run it with no since and the server can only mean "anything new from now on". The second scenario does that deliberately.

import asyncio, json, time
from urllib.parse import urlparse, parse_qs

HOLD_SECONDS = 0.5
events, waiters = [], []

def publish(text):
    events.append({"id": len(events) + 1, "text": text})
    for w in waiters:
        if not w.done():
            w.set_result(True)
    waiters.clear()

async def handle(reader, writer):
    line = (await reader.readline()).decode()
    while (await reader.readline()) not in (b"\r\n", b""):
        pass
    query = parse_qs(urlparse(line.split()[1]).query)
    # With no cursor the server can only mean "anything new from now on".
    since = int(query["since"][0]) if "since" in query else len(events)
    if not any(e["id"] > since for e in events):
        w = asyncio.get_running_loop().create_future()
        waiters.append(w)
        try:
            await asyncio.wait_for(w, HOLD_SECONDS)
        except asyncio.TimeoutError:
            pass
    body = json.dumps([e for e in events if e["id"] > since]).encode()
    writer.write(b"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\n"
                 b"Cache-Control: no-store\r\nContent-Length: " + str(len(body)).encode() +
                 b"\r\nConnection: close\r\n\r\n" + body)
    await writer.drain()
    writer.close()

async def get(port, path):
    r, w = await asyncio.open_connection("127.0.0.1", port)
    w.write(f"GET {path} HTTP/1.1\r\nHost: localhost\r\n\r\n".encode())
    await w.drain()
    raw = await r.read()
    w.close()
    return json.loads(raw.split(b"\r\n\r\n", 1)[1])

async def client(port, use_cursor, polls):
    t0, cursor, seen = time.monotonic(), 0, []
    for _ in range(polls):
        path = f"/updates?since={cursor}" if use_cursor else "/updates"
        batch = await get(port, path)
        ids = [e["id"] for e in batch]
        print(f"  t={time.monotonic() - t0:.1f}s got {ids if ids else 'nothing (timeout)'}")
        if batch:
            cursor = batch[-1]["id"]
            seen += ids
        await asyncio.sleep(0.2)            # the gap between polls
    return seen

async def scenario(use_cursor):
    events.clear()
    server = await asyncio.start_server(handle, "127.0.0.1", 0)
    port = server.sockets[0].getsockname()[1]
    async def updates():
        await asyncio.sleep(0.1); publish("CASE-1042 opened")
        await asyncio.sleep(0.1); publish("CASE-1042 assigned to Geetha")   # lands in the gap
        await asyncio.sleep(0.5); publish("CASE-1042 resolved")
    task = asyncio.create_task(updates())
    seen = await client(port, use_cursor, polls=4)
    await task
    server.close()
    await server.wait_closed()
    print(f"  published {[e['id'] for e in events]}, client saw {seen}")

for flag, label in [(True, "with a since cursor"), (False, "naive, no cursor")]:
    print(label)
    asyncio.run(scenario(flag))

Output on Python 3.13, the same across three runs:

with a since cursor
  t=0.1s got [1]
  t=0.3s got [2]
  t=0.7s got [3]
  t=1.4s got nothing (timeout)
  published [1, 2, 3], client saw [1, 2, 3]
naive, no cursor
  t=0.1s got [1]
  t=0.7s got [3]
  t=1.4s got nothing (timeout)
  t=2.2s got nothing (timeout)
  published [1, 2, 3], client saw [1, 3]

Update 2 was published at 0.2 s, while the client was between polls and no request was held. With a cursor, the next poll asked for everything after id 1, so the server answered at once with update 2. Without one, the next poll started from "now" and update 2 was gone for good, with no error on either side. The empty responses are the timeout path doing its job.

A few details decide whether this holds up in production:

• Keep the hold shorter than the shortest idle timeout on the path. Application Load Balancers and nginx both default to 60 seconds, so holds of 20 to 30 seconds are common. Otherwise the intermediary cuts the connection first.
• Send Cache-Control: no-store, or a proxy may replay an old answer to the next poll.
• Make the cursor a server-assigned increasing id, never the client's clock, and keep events long enough to cover a client that reconnects late.
• Back off with jitter after errors. Immediate re-polling is right after a success, but after a server restart every client retrying at once becomes a self-inflicted spike.
• Remember the HTTP/1.1 connection limit. Browsers typically allow about six connections per origin, so a held request in several tabs can starve page loads. HTTP/2 multiplexing removes that limit.
• Clean up waiters. This demo leaves timed-out futures in the list until the next publish. A real server removes them on timeout and on client disconnect.