Core concept

val df = spark.read.parquet("path").filter(...)
df.cache()
df.count()  // materializes
df.select(...)  // uses cached
Advertisement

How it works

Memory pressure. Choose storage level per data size + reuse pattern.

Advertisement

Trade-offs + gotchas

Iterative algorithms. Multi-action workflows.