Why it matters
Container sizing is where most YARN clusters leave performance on the table. Containers sized too small produce many concurrent containers per node that spend more time in overhead than work. Containers sized too large starve other applications and cause fragmentation where a big request cannot fit anywhere.
Understanding what a container costs — cgroup overhead, JVM overhead if applicable, localizer time — helps you size right. Guessing produces clusters that either underutilize hardware or drop tasks.
The architecture
The container resource specification has two mandatory fields: memory in megabytes and vcores. Memory is enforced by a Linux cgroup memory limit; a container that exceeds it is killed by the OOM killer. vcores are a Yarn abstraction meaning a fraction of a CPU core, enforced by cgroup CPU shares. A container with 2 vcores gets twice as much CPU as a container with 1 vcore when they compete.
Additional resources can be requested if configured: GPUs, FPGAs, disk. The resource types plugin lets administrators define custom resource types that the scheduler tracks. This is how YARN supports modern GPU workloads without breaking the memory-and-CPU baseline.
How it works end to end
When the AM requests a container, it specifies memory, vcores, node preferences, and priority. The scheduler picks a NodeManager with capacity and returns the allocation. The AM then sends a container launch request to the NM including the command to execute, environment variables, and any additional container resources.
The NM creates a cgroup with the requested limits and spawns the container's process tree inside it. Environment variables are set, working directory is created under the local NM directory, and any localized files are placed in the working directory. The container runs until it exits or is killed.
Container reuse is not native to YARN. Every task is a fresh container. Frameworks like Spark work around this by requesting long-lived executor containers that host many tasks, which amortizes the container startup cost.