Spark on Kubernetes architecture — operator, dynamic allocation, shuffle

Two front doors — spark-submit with a k8s master, or the operator CRD

There are exactly two ways a Spark application reaches a Kubernetes cluster, and the choice shapes everything downstream. The first is native spark-submit with a k8s:// master URL. In this mode spark-submit is a Kubernetes API client: it reads your kubeconfig, authenticates, and POSTs a single Pod object — the driver — into a namespace. The doubled scheme is not a typo. Everything after k8s:// is the API server URL, so k8s://https://api.example.com:6443 is correct; drop the inner scheme and Spark assumes HTTPS.

spark-submit \
  --master k8s://https://api.example.com:6443 \
  --deploy-mode cluster \
  --name nightly-rollup \
  --class com.example.Rollup \
  --conf spark.kubernetes.namespace=data-eng \
  --conf spark.kubernetes.container.image=registry.example.com/spark:3.5.1-app42 \
  --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark \
  --conf spark.executor.instances=20 \
  local:///opt/spark/app/rollup.jar

By default spark-submit then blocks, tailing the driver pod's status until it terminates. That is convenient from a laptop and wrong from an orchestrator that already owns its own timeout; spark.kubernetes.submission.waitAppCompletion=false makes submission fire-and-forget.

The second door is an operator. The Kubeflow Spark Operator defines a SparkApplication custom resource in the sparkoperator.k8s.io/v1beta2 group, plus a scheduled variant for cron-like runs. The operator is not a second scheduler and it does not reimplement Spark: it runs spark-submit on your behalf from inside its own pod, and it registers a mutating admission webhook that patches driver and executor pods with fields spark-submit historically could not express. What you gain is a declarative object with a status subresource, so kubectl get sparkapplications shows application state, and a restart policy that lives in the cluster rather than in whatever launched the job.

apiVersion: sparkoperator.k8s.io/v1beta2
kind: SparkApplication
metadata:
  name: nightly-rollup
  namespace: data-eng
spec:
  type: Scala
  mode: cluster
  image: registry.example.com/spark:3.5.1-app42
  mainClass: com.example.Rollup
  mainApplicationFile: "local:///opt/spark/app/rollup.jar"
  sparkVersion: "3.5.1"
  restartPolicy:
    type: OnFailure
    onFailureRetries: 2
  driver:
    cores: 2
    memory: "4g"
    serviceAccount: spark
  executor:
    instances: 20
    cores: 4
    memory: "12g"

Choose on ownership, not on taste. If Airflow or Argo already owns scheduling, retries and alerting, raw spark-submit is fewer moving parts and one less controller to upgrade. If you want Spark jobs to be first-class Kubernetes objects that GitOps reconciles and that RBAC can be written against, the CRD earns its keep. Both end up in the same place: a driver pod that creates executor pods.

What cluster mode actually creates in the namespace

Watch a namespace during a submit and four kinds of object appear. Knowing which is which turns most Spark-on-Kubernetes debugging into kubectl describe.

The driver pod. One pod, named after the application with a timestamp suffix and -driver, running a single container that executes the Spark driver JVM. This is the application. If it dies, the application is over — there is no controller that will bring it back in cluster mode, and an operator restart policy relaunches the whole job from the beginning rather than recovering the old one.

A ConfigMap. The resolved Spark properties are materialised into a ConfigMap and mounted into the pods, which is how configuration that was assembled on the submitting machine reaches processes inside the cluster rather than having to be re-specified there.

A headless Service. Executors need a stable address for the driver, so Spark creates a headless service whose selector matches only the driver pod, sets spark.driver.host to that service's cluster DNS name, and sets spark.driver.bindAddress to the pod IP. The driver's RPC port and its block-manager port are both exposed through it. This matters in client mode, where the driver is a pod you created yourself: you must build that headless service by hand, or executors will start, fail to reach the driver, and the job will hang with executors cycling.

Executor pods, each with an owner reference. Every executor pod carries an ownerReference pointing at the driver pod. That single field is Kubernetes' garbage collector doing your cleanup: delete the driver pod and every executor is reaped automatically. In client mode you get this only if you set spark.kubernetes.driver.pod.name to the name of the pod the driver is running in. Omit it and executors have no owner, so when the notebook pod holding the driver disappears, twenty executors keep running and billing.

One asymmetry surprises people at the end of a run: executor pods are deleted on termination, but the driver pod is not. It stays in Completed or Error because it holds the exit status and the last logs. The consequence is that a busy namespace accumulates dead driver pods until something prunes them.

RBAC and the driver service account — the first thing that fails

The driver creates its own executors. That means the driver, from inside the cluster, calls the Kubernetes API to create pods, and to create the service and ConfigMap it needs, and — if you use on-demand volumes — persistent volume claims. It authenticates with the service account token projected into its pod, and the default service account in a namespace can do none of those things. This produces the single most common first failure on Spark for Kubernetes, a driver that starts cleanly, requests its first executor and dies with Forbidden!Configured service account doesn't have access.

The Spark documentation's quick fix binds the built-in edit ClusterRole to the service account. It works and it grants vastly more than Spark needs. A namespace-scoped Role is a few more lines and is what belongs in a repository.

apiVersion: v1
kind: ServiceAccount
metadata: {name: spark, namespace: data-eng}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata: {name: spark-driver, namespace: data-eng}
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get", "list", "watch", "create", "delete", "patch"]
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get", "list"]
  - apiGroups: [""]
    resources: ["services", "configmaps", "persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata: {name: spark-driver, namespace: data-eng}
subjects: [{kind: ServiceAccount, name: spark, namespace: data-eng}]
roleRef: {kind: Role, name: spark-driver, apiGroup: rbac.authorization.k8s.io}

Two config families exist and they are routinely confused. spark.kubernetes.authenticate.driver.* configures the credentials the driver uses to talk to the API server — this is where serviceAccountName goes. spark.kubernetes.authenticate.submission.* configures the credentials spark-submit itself uses. Set the service account on the submission family and the job submits perfectly, then dies the moment the driver tries to create an executor.

The service account is also the anchor for cloud identity. On EKS an IRSA annotation on it, on GKE a workload-identity binding, is what lets executors read S3 or GCS without static keys baked into the image. Executors do not need Kubernetes API permissions, but they do need that cloud identity, which is why spark.kubernetes.authenticate.executor.serviceAccountName exists as a separate knob. The broader picture of credentials, encryption and audit is in Spark Security.

Executor pods have no controller behind them

This is the structural fact that explains most of the operational surprises. Spark's executor pods are bare Pod objects. There is no Deployment, no ReplicaSet, no StatefulSet, no Job. Nothing in Kubernetes knows those pods are supposed to number twenty. If a node dies and takes five executors, no controller replaces them — the driver notices and asks for more, and if the driver is gone, nothing does anything at all.

Inside the driver, an allocator loop reconciles a desired executor count against a snapshot of observed pod states. It builds that snapshot from two sources, and the redundancy is deliberate: a watch on pod events for low latency, plus a periodic full poll of the API server, governed by spark.kubernetes.executor.apiPollingInterval (30 seconds by default), because watches drop and a missed delete event would otherwise leave the driver believing in an executor that no longer exists.

Requests go out in batches rather than all at once — spark.kubernetes.allocation.batch.size (20 in current Spark) with a short delay between rounds, spark.kubernetes.allocation.batch.delay. That is not politeness. A job asking for 800 executors in a single burst hammers the API server's admission and scheduling path, and on a shared cluster the blast radius is everyone else's control plane latency, not just your job's startup time.

The behaviour to internalise is what happens when the cluster has no room. Spark asks for a pod, the scheduler cannot place it, the pod sits Pending. Spark does not give up and shrink the request, and it does not fail the job. It runs with whatever it got. So an under-provisioned cluster or an exhausted ResourceQuota produces a job that completes — hours late, at a fraction of the parallelism you configured — rather than an error. kubectl get pods showing a wall of Pending alongside a running driver is the signature, and kubectl describe pod on one of them names the reason: insufficient CPU, no matching node selector, an unsatisfied taint.

Two lifecycle flags are worth knowing before you need them. spark.kubernetes.executor.deleteOnTermination defaults to true, so failed executors vanish before you can inspect them — turn it off while debugging a crash and back on afterwards. And if you inject sidecars, spark.kubernetes.executor.checkAllContainers decides whether a dead sidecar counts as a dead executor; by default only the Spark container's status is considered.

Pod templates — the escape hatch for everything the config keys cannot say

The spark.kubernetes.* namespace covers a fixed vocabulary: labels, annotations, node selectors, environment variables, service accounts, volumes, resource requests and limits. Almost everything else a platform team cares about has no configuration key at all — tolerations, node and pod affinity, anti-affinity, init containers, sidecars, securityContext, topologySpreadConstraints, priorityClassName, imagePullSecrets beyond the simple form, DNS policy, host aliases.

Pod templates close that gap. You write an ordinary PodSpec YAML file, point spark.kubernetes.driver.podTemplateFile and spark.kubernetes.executor.podTemplateFile at it, and Spark uses it as the base object it then decorates. The file is read on the submitting side, so it must exist where spark-submit runs; with the operator you express the same thing inline in the custom resource instead.

apiVersion: v1
kind: Pod
spec:
  priorityClassName: batch-low
  tolerations:
    - key: workload
      operator: Equal
      value: spark
      effect: NoSchedule
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: karpenter.sh/capacity-type
                operator: In
                values: ["spot"]
  securityContext:
    runAsUser: 185
    fsGroup: 185
  containers:
    - name: spark-kubernetes-executor
      volumeMounts:
        - name: shuffle-scratch
          mountPath: /data/spark-local

The trap is precedence. Spark overwrites the fields it owns rather than merging them: the container image, the command and arguments, the resource requests and limits, the Spark-managed environment variables and volume mounts. Writing a memory limit into a template's resources block accomplishes nothing, because Spark computes and replaces it. Template fields Spark does not manage survive; fields it manages do not. Treat the template as a place for scheduling and security concerns, and keep resource sizing in Spark configuration where it belongs.

One more sharp edge in multi-container templates: Spark needs to know which container is the Spark one. spark.kubernetes.driver.podTemplateContainerName and its executor equivalent name it. Get it wrong, or leave a sidecar as the first container without setting it, and Spark decorates the wrong container — your log-shipping sidecar receives the Spark command line and the actual executor never starts.

Requests, limits, memory overhead and the OOMKilled you will meet

Memory on Kubernetes behaves differently from memory on YARN in one decisive way, and Spark's configuration reflects it. For Spark pods, the container's memory request and limit are set to the same value. There is no spark.kubernetes.executor.limit.memory; memory for a Spark pod is not burstable. That value is the sum of spark.executor.memory, spark.executor.memoryOverhead, spark.memory.offHeap.size, and spark.executor.pyspark.memory when set.

Overhead is computed, not guessed: spark.executor.memoryOverheadFactor times the executor memory, subject to a floor. It defaults to 0.10 — except for non-JVM jobs on Kubernetes, where it defaults to 0.40, because PySpark workers are separate processes living entirely outside the heap and the 10% default was a reliable way to get killed.

Now the failure mode. The JVM heap is sized from spark.executor.memory alone. Everything else in that container comes out of the overhead: JVM metaspace, thread stacks, the code cache, Netty's direct byte buffers for shuffle transfer, Python worker processes, native libraries, and anything written to a RAM-backed scratch directory. When the container's total crosses the limit, the kernel's OOM killer takes the process. The pod goes Terminated with reason OOMKilled and exit code 137. There is no OutOfMemoryError in the log, no heap dump, and often no Spark-level error at all — just an executor that stopped mid-task.

That distinction is the most useful diagnostic on the platform. A Java OutOfMemoryError in the executor log means the heap was too small: raise spark.executor.memory or reduce per-task memory pressure. An OOMKilled with no Java exception means the container was too small for everything around the heap: raise spark.executor.memoryOverhead. Raising executor memory in response to an OOMKilled raises the overhead too (it is a factor), which is why the wrong fix sometimes appears to work and then fails again at a larger scale.

CPU is the mirror image, because CPU is burstable. spark.executor.cores is a Spark concept — the number of task slots — and Kubernetes never sees it. spark.kubernetes.executor.request.cores sets the actual CPU request, and it accepts fractional and millicore values like 500m, while spark.kubernetes.executor.limit.cores sets the limit. Requesting less than spark.executor.cores packs more executors per node and is a legitimate overcommit strategy for bursty workloads; leaving the limit unset lets an executor use idle node capacity. Setting a tight limit gets you CFS throttling, which shows up as tasks that are inexplicably slow while CPU metrics look unremarkable. How the memory inside the executor is then divided between execution and storage is Spark unified memory management, and sizing tasks against slots is Spark stages and tasks.

Dynamic allocation without an external shuffle service

Start with the blunt fact, because a great deal of confused advice exists on this point: Spark on Kubernetes does not support the external shuffle service. Setting spark.shuffle.service.enabled=true is not a valid path here — the Spark documentation says so explicitly in its dynamic-allocation caveats. There is no NodeManager-equivalent daemon on each node to hold shuffle files after the executor that wrote them is gone. What that service does on YARN, and why decoupling shuffle from executor lifetime matters at all, is the external shuffle service architecture; the point here is that you do not get it.

The consequence is direct. An executor's shuffle output lives in that executor's local directories and is served by that executor's block manager. Remove the executor and the blocks are gone; any later stage that needs them triggers a FetchFailedException, and the scheduler re-runs the map stage that produced them. Dynamic allocation, whose entire job is to remove idle executors, therefore becomes a machine for destroying work unless you configure one of three supported alternatives.

Shuffle tracking is the cheap and common one. spark.dynamicAllocation.shuffleTracking.enabled=true makes the driver keep account of which executors hold shuffle data for shuffles that are still referenced, and refuse to scale those executors down. It costs nothing to enable and it is what most Kubernetes deployments run. The price is that executors pinned by shuffle data sit idle and billed, so a job with a long tail after a big shuffle holds a wide cluster while doing very little. spark.dynamicAllocation.shuffleTracking.timeout puts a ceiling on how long an executor is protected, trading a risk of recomputation for release.

Graceful decommissioning is the one that actually lets executors leave. Set spark.decommission.enabled=true together with spark.storage.decommission.shuffleBlocks.enabled=true and a departing executor migrates its shuffle blocks — and, with spark.storage.decommission.rddBlocks.enabled, its cached RDD blocks — to surviving peers before it exits. When no peer has room, spark.storage.decommission.fallbackStorage.path gives it an object-store destination. Note the documented caveat on that path: Spark never cleans it up, so it needs a bucket lifecycle rule or it grows forever.

A reliable shuffle plugin is the third: spark.shuffle.sort.io.plugin.class swaps in a ShuffleDataIO implementation that writes shuffle data to remote storage, which is the seam that remote-shuffle-service projects plug into. It is the most complete answer and the most infrastructure to run.

Everything about when the allocator adds or removes executors — backlog timeouts, idle timeouts, the ramp — is platform-independent and is covered in Spark dynamic allocation architecture. Only the shuffle-preservation half of the story is Kubernetes-specific.

Local disk, emptyDir and PVC reuse — where shuffle files actually live

Configure nothing and Spark mounts an emptyDir for spark.local.dir. An emptyDir is a directory on the node's own filesystem, which on a managed node pool is usually the root disk — often modest, and shared with the container runtime's image store and every other pod's logs. A shuffle-heavy or spill-heavy job will fill it. What happens next is worse than a failed job: the kubelet raises DiskPressure on that node and starts evicting pods, which may not be yours. Sizing local storage is a neighbourly obligation on Kubernetes in a way it never was on a dedicated YARN cluster. What generates that traffic — sort spill, map output files, the fetch side — is Spark shuffle architecture.

spark.kubernetes.local.dirs.tmpfs=true backs the scratch directory with RAM instead. It is genuinely faster, and it charges every spilled byte against the pod's memory limit, converting a disk-space problem into an OOMKilled. It is a reasonable choice only for jobs whose shuffle is known to be small and whose memory overhead has been raised to cover it.

The durable answer is a volume per executor. Spark's volume configuration mounts a persistent volume claim into executor pods, and the special claim name OnDemand tells it to create a fresh PVC for each executor from a storage class you name.

spark.kubernetes.executor.volumes.persistentVolumeClaim.scratch.options.claimName   = OnDemand
spark.kubernetes.executor.volumes.persistentVolumeClaim.scratch.options.storageClass = fast-ssd
spark.kubernetes.executor.volumes.persistentVolumeClaim.scratch.options.sizeLimit   = 500Gi
spark.kubernetes.executor.volumes.persistentVolumeClaim.scratch.mount.path          = /data/spark-local
spark.kubernetes.executor.volumes.persistentVolumeClaim.scratch.mount.readOnly      = false
spark.local.dir                                                                     = /data/spark-local

Then comes the part with no equivalent on any other cluster manager. spark.kubernetes.driver.ownPersistentVolumeClaim makes the driver, not the executor, the owner of those on-demand claims, so a PVC survives the death of the pod that was using it. spark.kubernetes.driver.reusePersistentVolumeClaim then hands an orphaned PVC to the next executor Spark creates. Both default to true in current Spark, but they only do anything if you have configured on-demand PVCs in the first place — which is the step people skip.

The payoff is real: when a spot reclaim kills an executor, the shuffle files it had written are still sitting on a volume, and the replacement executor mounts it and can serve them. It is the closest thing Kubernetes offers to shuffle survival, and it is why PVC reuse and spot instances belong in the same conversation. Two caveats before you rely on it. Block storage is zonal on every major cloud, so a reused PVC pins its replacement pod to one zone and can starve if that zone has no capacity. And executor churn becomes storage-controller churn — attach and detach operations are slow and rate-limited, so a job that cycles executors quickly can spend more time waiting on volume attachment than it saves in recomputation.

Placement — node selectors, taints, spot pools and gang scheduling

The default Kubernetes scheduler will happily put your driver anywhere, which is the first thing to fix. spark.kubernetes.node.selector.[labelKey] constrains both roles; the separate spark.kubernetes.driver.node.selector.[labelKey] and spark.kubernetes.executor.node.selector.[labelKey] families let you split them, and that split is the standard production shape: driver on a stable on-demand pool, executors on whatever is cheap.

Anything richer than a selector — tolerations for a tainted Spark-only node pool, anti-affinity to spread executors across nodes so a single node loss does not take a quarter of your shuffle output, topology spread constraints across zones — has no configuration key and belongs in a pod template.

Spot capacity is where this all matters. An executor killed by a spot reclaim is, to Spark, simply a lost executor: its shuffle data is gone and its running tasks are retried elsewhere. That is survivable and often economically obvious. A driver killed by a spot reclaim ends the application, and no restart policy recovers the work already done — the operator relaunches from the start. So the rule is unambiguous: never put the driver on preemptible capacity. Beyond that, two things make executor loss cheap. Enable decommissioning so that the pod's termination grace period is spent migrating blocks rather than being idle, and pair it with a node-termination handler that turns the cloud provider's reclaim notice into a graceful pod deletion. Without the handler, the notice expires and the node is simply cut off, which converts a graceful decommission into a hard loss.

The last placement problem is one YARN solved for you. Kubernetes schedules pods one at a time and has no concept of a job. Two Spark applications each requesting 50 executors on a cluster with room for 60 will each get roughly half, and both will run slowly while neither can finish and release. There is no queue and no fair-share arbiter in the default scheduler. Barrier-mode workloads make it fatal rather than merely wasteful, because they require every task to be running simultaneously — see Spark barrier execution mode.

The fix is a batch scheduler. spark.kubernetes.scheduler.name directs driver and executor pods at an alternative scheduler instead of the default one, and the two with real Spark integration are Volcano and Apache YuniKorn. Both add the missing primitives: gang scheduling, so a pod group is placed all-or-nothing, and hierarchical queues with capacity guarantees and preemption. If you are moving a multi-tenant workload off YARN, this is the piece you must replace deliberately, because nothing in a stock cluster does it.

Images and shipping dependencies into the pod

On YARN the cluster already has Spark installed. On Kubernetes the image is the installation, and the version inside it must match what you submit with. Spark ships Dockerfiles in the distribution and a helper to build from them — bin/docker-image-tool.sh -r registry.example.com -t 3.5.1 build push, with -p selecting the PySpark Dockerfile and -R the R one. Driver and executor images can differ via spark.kubernetes.driver.container.image and its executor counterpart, but they rarely should; a version skew between them produces serialization errors that name nothing useful.

Getting application code and dependencies in has three routes, in decreasing order of reliability.

Bake them into the image and reference them with the local:// scheme, which means "already present in the container filesystem" — local:///opt/spark/app/rollup.jar. Startup is instant, the artifact is immutable and reproducible, and the job has no runtime dependency on a file server. This is the right default, and the friction people cite against it — a rebuild per code change — is a CI problem, not an architecture problem.

Fetch from shared storage by giving object-store paths for the main application file, --jars or --files. Every executor fetches independently, so a 200 MB dependency across 300 executors is 60 GB of reads concentrated in the first seconds of the job, and a thundering herd against one prefix is a genuine way to get throttled.

Let spark-submit upload them. Setting spark.kubernetes.file.upload.path to a Hadoop-compatible location makes spark-submit stage client-side file:// dependencies there for the pods to fetch. It is convenient for iteration and has two documented sharp edges: everything lands in a flat directory, so two dependencies sharing a basename silently overwrite each other, and the submitting machine needs both credentials and the filesystem implementation jars for that path.

A word on --packages. Ivy resolution needs a writable Ivy home, and Spark images typically run as a non-root user with a home directory that is read-only or absent — hence the -Divy.cache.dir=/tmp -Divy.home=/tmp incantation you see in every example. Beyond the mechanics, resolving from Maven Central at job start puts a public network dependency on the critical path of your production pipeline. Resolve once, bake the resulting jars into the image, and delete the flag.

Finally, registry details bite. spark.kubernetes.container.image.pullSecrets names the secrets for a private registry, and the pull policy defaults to IfNotPresent, meaning a mutable tag like :latest can leave different nodes running different code in the same application. Tag immutably by content or build number.

Logs, the vanished driver pod, and what you keep after the job ends

kubectl logs does not read from a log store. It reads the container's output through the kubelet on the node where the pod ran, so the moment the pod object is deleted — or the node is scaled away — the logs are gone with it. Combine that with spark.kubernetes.executor.deleteOnTermination defaulting to true and you get the characteristic Kubernetes failure of a data platform: someone reports that last night's job failed, you go to look at the executor that crashed, and there is nothing there at all.

Three defences, and you want all three. While actively debugging, set spark.kubernetes.executor.deleteOnTermination=false so failed executor pods stay in Terminated state and can be read and described — then turn it back on, because leaked pods consume namespace quota and count against the per-node pod limit. Permanently, run a cluster-level log agent that tails container logs off the node into a store; that is the only mechanism that survives pod deletion, and it is infrastructure, not Spark configuration. And enable the event log: spark.eventLog.enabled=true with spark.eventLog.dir pointed at object storage, read back by a History Server.

Do not conflate those last two. Container logs are stdout and stderr — exceptions, stack traces, the reason a JVM died. The event log is a structured record of every job, stage, task and SQL plan, and it is what you need to diagnose skew, a bad join strategy or a ragged last wave after the fact. The live Spark UI runs inside the driver pod on port 4040 and dies with it, so during a run you reach it with kubectl port-forward or an ingress, and afterwards only the History Server has the data.

Pod-level signals are the other half. Exit code 137 is a SIGKILL: on a Spark pod that almost always means OOMKilled, and the pod's lastState.terminated.reason confirms it. Exit code 143 is a SIGTERM — the pod was told to stop, which means eviction, preemption, a node drain or a spot reclaim, and not a Spark failure at all. Reading those two apart saves hours, because a job full of 143s is a capacity story and a job full of 137s is a memory story, and the fixes have nothing in common.

For metrics, Spark's metrics system can expose executor metrics in Prometheus format when spark.ui.prometheus.enabled=true, and its Dropwizard sinks are configured through a metrics.properties baked into the image. Because the scrape target is a pod and pods here are ephemeral, this works best when your Prometheus discovers pods by label and you have set spark.kubernetes.driver.label.[name] and the corresponding annotations so every application is discoverable the same way.