Why architecture matters here
The question Cats Effect answers is the same one every async runtime answers — how do you run enormous numbers of concurrent activities safely on few threads — but its architectural stance is distinctive in two ways that matter in production. First, laws over conventions: the behavior of cancellation, error handling, and resource release is specified by typeclass laws that every instance must pass. When you write resource.use(f).timeout(2.seconds), the guarantee that the release action runs exactly once — whether f succeeded, failed, or was cancelled by the timeout — is not a documentation promise but a property checked by the laws suite. Teams inherit correctness they did not have to design.
Second, abstraction as an ecosystem contract: http4s routes, fs2 pipelines, and doobie transactors are written against Async[F], not IO. That indirection sounds academic until the day you need to run the same code under a test effect with controllable time, embed it in another runtime, or thread context (tracing spans, request ids) through Kleisli — then it is the difference between a refactor and a rewrite. The concrete runtime remains IO's, and knowing its mechanics is what operational excellence looks like: why a blocking JDBC call must be IO.blocking (it shifts to the unbounded blocking pool instead of parking a compute worker), why a hot CPU loop needs IO.cede or the auto-yield to keep latency fair, why timers scale (each worker thread owns a timer heap — no global scheduler contention at 100k sleeping fibers).
The starvation checker deserves its own mention: CE3 ships with a canary that measures scheduler responsiveness and prints a warning when compute workers are starved — turning the classic invisible failure of async runtimes ('everything is slow, nothing is busy') into an explicit, documented diagnosis. Architecture that surfaces its own pathologies is architecture you can operate.
The architecture: every piece explained
Top row: from description to execution. IO[A] is an immutable tree of nodes — Pure, Delay, FlatMap, Async, Sleep, Uncancelable — built by your program. The IORuntime bundles the machinery: the compute pool, the blocking pool, the scheduler, and runtime config. Calling unsafeRunSync/IOApp.run creates a root fiber — a run loop object that interprets the tree, holding its continuation stack and cancellation state. Fibers are heap objects costing ~hundreds of bytes; start, both, race, and fs2's concurrency operators fork them freely.
The work-stealing pool is CE3's crown jewel. One worker thread per core, each with a local queue; overflow and wake-ups handled via an external queue; idle workers steal from busy ones. Two integrations make it unusual. Timers live in the workers: IO.sleep registers in the owning worker's timer heap, so a million sleeping fibers cost no dedicated scheduler thread and no cross-thread handoff on wake — crucial for timeout-heavy servers. Fairness is automatic: the run loop counts operations and cedes the worker after a budget (default 1024), so a long chain of cheap flatMaps cannot monopolize a core; explicit IO.cede handles the pathological pure-compute cases. IO.blocking transfers the fiber to the blocking pool (cached, unbounded) and back, keeping compute workers hot; IO.interruptible marks blocking regions that may be interrupted via thread interruption.
Middle row: semantics. Cancellation is cooperative and scoped: cancellation is checked at interpreter boundaries, uncancelable opens a protected region, and Poll re-enables cancellation inside it for the waiting parts — the idiom that makes acquire-then-wait patterns safe. Resource[F, A] composes acquire-release pairs monadically: build a whole application as one Resource (config → pools → clients → server), and teardown runs in reverse order on exit or failure. The typeclasses stack capabilities: Sync (suspend side effects), Async (integrate callbacks), Concurrent (fibers, Ref, Deferred), Temporal (time); libraries request the minimum they need. Ref (atomic mutable cell) and Deferred (single-shot promise) are the coordination bedrock from which queues, semaphores, and fs2's machinery are built.
Bottom rows: the ecosystem rides on these guarantees — fs2 turns pull-based streaming with resource safety into a library; http4s expresses servers as Kleisli[F, Request, Response]; doobie wraps JDBC in typed, bracketed programs. Tracing reconstructs logical stack traces across async boundaries (cached at construction, nearly free), and fiber dumps plus the starvation checker fill the ops strip.
End-to-end flow
Boot and serve one request in a typical http4s service. IOApp assembles the app as a Resource: load config (IO.blocking file read), build a doobie transactor with its own bounded JDBC pool, an http4s Ember client with connection pooling, then the Ember server binding port 8080. resource.useForever acquires each in order; teardown is registered in reverse. The runtime starts: N compute workers, timer heaps empty, blocking pool idle.
A request hits an endpoint that aggregates a database row with two downstream API calls under a deadline: (dbLookup, apiA, apiB).parTupled.timeout(800.millis). The route's fiber forks three children. dbLookup enters doobie: the JDBC call runs under IO.interruptible on the blocking pool; the compute worker that launched it immediately dequeues other work. The two HTTP calls are genuinely async: NIO callbacks re-enter the runtime via IO.async, waking fibers on whichever worker is free. The timeout registered a timer in the parent worker's heap.
apiB is slow today. At 800ms the timer fires on its owning worker, cancelling the parTupled fiber; cancellation propagates to all three children. dbLookup already completed (its connection was returned by doobie's bracket); apiA's fiber is cancelled between retries — its in-flight socket is released by the client's resource management; apiB's fiber is cancelled at its next interpreter boundary. The route's recoverWith maps the TimeoutException to a 504 with a degraded payload. Every finalizer ran; nothing leaked; the reconstructed trace on the error shows the logical chain — route → aggregate → apiB — across async hops.
Under load, the fairness machinery earns its keep: a batch endpoint computing large JSON encodings cedes every 1024 ops, so p99 latency on the cheap health endpoint stays flat. One day someone adds a legacy SDK call without IO.blocking; within minutes the starvation checker prints its warning naming the symptom, a fiber dump shows compute workers parked in the SDK, and the fix is one combinator. On SIGTERM, IOApp cancels the main fiber: the server stops accepting, in-flight requests drain within the grace period, then client, transactor, and pools close in reverse — shutdown as the same composed value as startup.