Why architecture matters here
Scheduler architecture matters because it directly affects tail latency, fairness, and throughput. A scheduler that isn't NUMA-aware ping-pongs threads and destroys cache; a scheduler that lacks priority discipline lets background jobs steal cycles from interactive workloads. Tuning it is often the difference between p99 5ms and 50ms.
Cost impact is real. Right scheduling policy + affinity can free 20-30% of throughput on cache-sensitive workloads.
Reliability under load is where the scheduler shines when tuned right and hurts when not. Real-time classes for critical work, cgroups for isolation, cpusets for affinity keep behavior predictable.
The architecture: every layer explained
Walk the diagram top to bottom.
Runnable Tasks. Threads and processes ready to run. Each has state, priority, and history.
Scheduler. The kernel component that picks next task per CPU. Runs on every timer tick + on wakeups.
CPUs. Physical cores + hyperthreads. Scheduling accounts for both.
CFS (Linux default). Completely Fair Scheduler tracks virtual runtime per task; the task with the lowest vruntime runs next. Red-black tree keeps sorted access.
Priority + Niceness. nice values (-20 to +19) skew vruntime accumulation. Priorities via classes (real-time, batch, idle).
Work Stealing. User-space (Go runtime, Java ForkJoin, Rust Tokio) implements its own scheduling on top of kernel threads. When a worker idles, it steals from busy workers' queues.
Preemption. Kernel preempts a running task when timer fires or higher-priority task wakes.
Real-time. SCHED_FIFO (run until yield/higher priority) or SCHED_RR (round-robin at same priority). Use carefully; can starve normal work.
cgroups + Kubernetes. Container isolation. CPU limits, requests. Kubernetes maps to cgroups.
NUMA + affinity. Threads run near their memory. sched_setaffinity() pins to specific CPUs.
End-to-end scheduling event flow
Trace a scheduling event. Application has 100 threads running on 8 CPUs.
Timer tick fires. Kernel enters scheduler on CPU 0. Current task's vruntime is updated (add time consumed × 1024 / weight). Kernel checks: is there a task with lower vruntime?
CFS looks at leftmost node of red-black tree for CPU 0's run queue. Selects. Context switch: save current registers, load new task's registers, switch TLB if needed.
Higher-priority thread (say, real-time) wakes elsewhere. Kernel preempts even mid-quantum. Real-time thread runs.
Meanwhile Go program has 1000 goroutines but only 8 kernel threads. Go's user-space scheduler distributes goroutines across those threads with work stealing. Idle worker steals from busy worker's queue.
CPU 4 is idle. Its scheduler checks for tasks. Local run queue empty. Load balancer moves a task from CPU 2 (loaded) to CPU 4. Cache warm-up cost paid; balance restored.
NUMA-aware scheduling: task with memory on NUMA node 0 preferred on CPUs 0-3 (co-located). Moving it to CPUs 4-7 (other NUMA node) would incur remote memory access.
Run queues, time slices, and what a switch really costs
Every CPU owns a private run queue. That is the first architectural decision worth understanding: there is no single global list of runnable threads, because a global list needs a global lock, and a global lock on a 64-core machine serialises the one operation that runs on every core thousands of times a second. Each core picks from its own structure under its own lock, and cross-core movement becomes an explicit, comparatively rare balancing act rather than the default path.
A time slice is not a fixed constant either. CFS worked from a target latency - the window in which every runnable task on a core should get the CPU at least once - and divided it by the number of runnable tasks, weighted by priority. Ten runnable threads sharing a 24 ms window get roughly 2.4 ms each. A hundred runnable threads would get 240 microseconds each, so a minimum-granularity floor clamps the slice and the window stretches instead. Under CFS these were exposed as sched_latency_ns and sched_min_granularity_ns; EEVDF reworked part of that knob set, so read the values on the kernel you actually run rather than copying a tuning guide.
The slice length matters because a switch is not free. The direct cost - save registers, swap stacks, update the page-table pointer - is on the order of a microsecond. The cost you actually pay is the cold cache and cold TLB the incoming thread lands in: its working set was evicted while it waited, and it now stalls on memory until the caches refill. On a memory-heavy workload that indirect cost is easily an order of magnitude larger than the switch itself, which is why running far more threads than cores hurts even when every thread is doing useful work.
Fair share versus strict priority
Two different disciplines coexist in one kernel, and it pays to be precise about which one a given behaviour comes from.
CFS is proportional share, not priority. Each task accumulates virtual runtime: real time consumed, scaled by the inverse of its weight. The scheduler runs whichever task has the least virtual runtime, so a heavier task accumulates vruntime more slowly and therefore gets picked more often. Nice values map onto that weight table - nice 0 is weight 1024, and each nice step is roughly a 1.25x multiplier - so nice -5 gets several times the share of nice 0 when both are runnable. A nice +19 task is never forbidden from running; it just gets a thin slice of a contended core. EEVDF changed the selection rule rather than the goal: a task must be eligible, meaning it has not yet consumed its fair share of elapsed time, and among eligible tasks the earliest virtual deadline wins. That split lets a task ask for shorter, more frequent slices to improve its latency without asking for more total CPU, which pure lowest-vruntime selection could not express.
The real-time classes are a different mechanism entirely. SCHED_FIFO and SCHED_RR sit in a strictly higher class: any runnable real-time task preempts every normal task on that core, whatever its nice value. A SCHED_FIFO thread that spins forever will wedge a core, which is why the kernel ships a throttle that caps real-time tasks at 950 ms of each second and leaves the rest for everything else. Disabling that throttle on a production box is how you build a machine you cannot log into.
Strict priority also brings priority inversion, where a high-priority task blocks on a lock held by a low-priority task that a mid-priority task keeps preempting. That failure mode and its fixes - inheritance and ceiling protocols - are covered in priority inversion architecture and are not repeated here.
Wakeup placement, affinity, and NUMA locality
Most scheduling decisions are not "who runs next" but "where does this waking thread go". When a thread becomes runnable the kernel chooses a target CPU before it ever enters a run queue, and that choice is where cache locality is won or lost. The heuristic favours the CPU the thread last ran on, since its cache may still be warm, and for a wakeup triggered by another thread it favours a CPU near the waker, since the data the waker just touched is in that cache. If the preferred CPU is busy, the search widens to an idle sibling in the same cache domain before ranging further.
Those domains are hierarchical and the hierarchy mirrors the hardware: two hyperthreads share L1 and L2, cores on a die share L3, sockets share only the interconnect. Migrating within L3 costs a little. Migrating across sockets costs a lot, because the thread's memory is still allocated on the old node. Remote memory access is not catastrophic on its own - roughly 1.5x to 2x local latency on a typical two-socket machine - but a thread that migrates every few milliseconds pays it continuously and never rebuilds a warm cache.
Two counter-measures exist. Automatic NUMA balancing samples memory accesses and either migrates pages toward the node running the thread or migrates the thread toward the node holding the pages; it is a background heuristic and it spends page faults to do its sampling. Explicit placement is the other route: taskset and sched_setaffinity() pin a thread to a CPU mask, cpusets do the same for a whole cgroup, and numactl --membind ties allocation to a node. Pinning is powerful and easy to misuse - pinning latency-critical threads onto CPUs that still carry interrupts, timers, and general kernel work gives you the worst of both worlds. Serious low-latency setups isolate those CPUs from the general balancer first, then pin onto them.
cgroup quota: the throttling trap behind p99 spikes
Containers do not receive a slice of CPU. They receive a budget with a deadline, and that difference produces one of the most common tail-latency mysteries in production. Under cgroup v2, cpu.max is a quota plus a period, and the period defaults to 100 ms. A container allowed 0.5 CPU may consume 50 ms of CPU time in each 100 ms window. When the budget is gone the entire cgroup is throttled: every thread in it is dequeued and nothing runs until the period rolls over.
The failure mode follows directly from the arithmetic. A request needing 20 ms of CPU arrives 60 ms into a window with 10 ms of budget left. It runs for 10 ms, sits frozen for 40 ms, then finishes. Average CPU utilisation reads about 50 percent - nowhere near the limit - while p99 shows a 40 ms cliff that profiling the application will never explain, because the application was not running. Multithreaded runtimes make it sharper: eight busy threads burn a 50 ms budget in 6 ms of wall time, and the container is then dark for 94 ms.
The evidence lives in cpu.stat, in the nr_throttled and throttled_time counters. If nr_throttled is climbing, the limit is the problem and the code is not. The fixes are to raise the quota, to reduce in-container parallelism so the runtime stops spending the whole budget in a burst - a runtime that sizes its thread pool or GC threads from the host core count while running under a fractional quota is the usual culprit - or to drop the hard limit entirely and rely on proportional weights for contention instead.
User-space scheduling and the M:N model
A user-space scheduler multiplexes many logical tasks onto few kernel threads: M tasks over N carriers, where N is typically the core count. The kernel schedules the carriers, the runtime schedules tasks onto carriers, and both layers are making placement decisions about the same work without visibility into the other's intent. That two-level structure is what makes user-space scheduling both cheap and prone to surprising stalls.
How a task suspends is the coroutine model, covered in coroutines, and for the JVM's continuation and carrier implementation in Java virtual threads. The scheduling question is the complementary one: how does the runtime ever get control back?
Cooperative: control returns only when the task gives it up
In a cooperative runtime every await point is a yield, and a task that computes for a long stretch without awaiting owns its carrier for that entire stretch. Every other task queued on that carrier waits. This is not a defect, it is the definition of cooperative, and it is why runtimes add artificial yield points: Tokio gives each task a budget of operations, after which its resource calls start reporting "not ready" specifically to force a yield. That is a cooperative scheduler manufacturing a yield the task declined to provide.
Preemptive: control is taken back
Go originally could only preempt at function-call safepoints, so a tight arithmetic loop with no calls in it could hold a processor indefinitely and stall the whole runtime. Go 1.14 added asynchronous preemption: a monitor thread signals the running thread, and the signal handler parks the goroutine at a safe point. The price is that the runtime must be able to unwind and describe its stack at arbitrary instruction boundaries, which is exactly the complexity cooperative runtimes are avoiding.
Load balancing between workers: stealing seen from above
A runtime with per-worker queues needs some way to move work when one worker is idle and another has a backlog, and the standard answer is work stealing. The mechanics - the per-worker deque, which end the owner uses and which end thieves take from, and why that split keeps the common case uncontended - are covered in depth in work-stealing scheduler architecture and work-stealing schedulers.
What belongs to the scheduler view is the interaction between layers. When a thief takes a task, that task's data is warm in the victim's cache and cold in the thief's, so a steal trades a locality miss for a utilisation win - the right trade only when the victim genuinely has a backlog. Meanwhile the kernel's own balancer may migrate the worker threads themselves between cores for reasons the runtime knows nothing about, so a task can move workers and then have its worker move cores, paying twice.
That makes steal rate a health metric rather than an implementation detail. Near-zero steals while workers sit idle means work is not reaching them, usually because it is all being submitted through one queue. A very high steal rate on a machine that is not saturated usually means the tasks are too small, and scheduling overhead now exceeds the work being scheduled.
Blocking calls poison a scheduler
A scheduler can only place work it knows about. Any blocking operation the runtime cannot observe removes a carrier from service without telling anyone, and the damage scales inversely with how few carriers there are: lose one of eight and you have lost an eighth of the machine.
Syscalls are the visible case, and runtimes compensate. A goroutine entering a syscall parks its OS thread; a monitor thread notices a processor sitting in a syscall and hands that processor to another thread so the remaining goroutines keep running. Java's ForkJoinPool exposes the same idea explicitly through ManagedBlocker: a task about to block declares it, and the pool starts a compensation thread so the target parallelism is maintained. The JVM's virtual threads unmount cleanly on blocking I/O but pin the carrier inside synchronized blocks, which converts a lightweight park into a genuinely lost carrier.
The invisible cases are the dangerous ones. A major page fault blocks with no runtime involvement at all, including a read from an mmap'd file that looks to the code like an ordinary memory access. A blocking name resolution, a native call into a library that does its own I/O, and a spin on a mutex whose holder has been descheduled all have the same shape: N carriers, one of them gone, no compensation, no log line.
The counter-measures are structural. Keep known-blocking work off the carriers entirely, on a separate pool sized for waiting rather than for cores - that sizing question belongs to thread pools. Prefer synchronisation primitives the runtime understands over ones it does not. And where compensation threads exist, bound them, because an unbounded compensation policy turns one slow dependency into thousands of OS threads and a machine that is now genuinely out of memory rather than merely slow.
Observability: measure waiting, not usage
Scheduling problems present as latency with no CPU-bound smoking gun, so the metrics that matter measure time spent runnable-but-not-running.
# run queue depth: the 'r' column counts runnable, not blocked
vmstat 1
# voluntary vs involuntary switches; nvcswch/s means preempted, not yielding
pidstat -w -p <pid> 1
# how long runnable tasks wait for a CPU, as a microsecond histogram
runqlat 10 1
# share of wall time some task was runnable but starved of CPU
cat /proc/pressure/cpu
# per-container throttling evidence
grep -E 'nr_throttled|throttled_usec' /sys/fs/cgroup/<path>/cpu.stat
# per-task run time, wait time, and slice count
cat /proc/<pid>/schedstatRead them together. Run-queue depth sustained above the core count means more runnable work than cores, which is saturation whatever the utilisation number says. A high involuntary switch rate means threads are being preempted mid-work rather than yielding, the signature of too many runnable threads rather than slow ones. Run-queue latency is the number that maps directly onto user-visible delay: if that histogram's tail sits in milliseconds, those milliseconds are added to requests, and no application profiler will show them because the thread was not executing. PSI's cpu some avg10 expresses the same loss as a percentage.
Two traps are worth naming. Linux load average counts uninterruptible-sleep tasks, so a host with a stalled disk shows a load of 40 alongside idle CPUs - load average is not run-queue depth and never was. And per-process CPU time says nothing about scheduling delay: a thread that took 200 ms of wall clock for 5 ms of CPU spent 195 ms somewhere, and the run time versus wait time split in schedstat is what tells you whether it was waiting for a CPU or waiting for something else entirely.
Which knob, in what order
Order matters, because most scheduler tuning makes things worse.
Start by reducing the number of runnable threads. Almost every reported scheduler problem is an application with far more runnable threads than cores, and the fix is less parallelism, not a different policy. Next check for throttling: nr_throttled is a two-second check that explains a large share of containerised latency mysteries. Then check for the sizing mismatch - a runtime that reads the host core count while running under a fractional quota produces both the throttling and the excess switching at once.
Only after that should policy change. Nice values and cgroup weights are the safe first move, because proportional sharing degrades gracefully: a deprioritised task gets slower, it does not stop. Affinity and CPU isolation come next, and only when you have a small number of latency-critical threads and are prepared to dedicate cores to them and keep everything else off. SCHED_FIFO is last, justified only when a missed deadline is a real failure rather than an annoyance, and then only with the real-time throttle left enabled and a watchdog in place, because the failure mode of a runaway real-time thread is a machine nobody can reach.
The framing behind that ladder is that fairness, throughput, and latency are three objectives in tension. Throughput wants long slices - fewer switches, warmer caches, less balancing. Latency wants short slices and eager preemption so a freshly woken task runs now rather than after someone else's quantum. Fairness wants equal progress, which can cost both by preempting a task that was one millisecond from finishing. No configuration maximises all three, so the productive move is to declare which one the machine exists for and let the scheduling classes express it: SCHED_BATCH for work that should stop receiving wakeup preference, SCHED_IDLE for work that should run only when nothing else wants the CPU.