Why it matters

Caching wrong data wastes memory. Missing caches wastes compute. Understanding both matters.

Advertisement

The architecture

cache(): MEMORY_ONLY default.

persist(StorageLevel): configurable.

Storage levelsMEMORY_ONLYfast, drops on pressureMEMORY_AND_DISKspills to diskDISK_ONLYfor huge dataSerialized levels save memory at CPU cost; useful for large tables
Common levels.
Advertisement

How it works end to end

MEMORY_ONLY_SER: serialized. Less memory, more CPU.

Off-heap: OFF_HEAP for very large caches with GC pressure.

Unpersist: release when done. Important for long jobs.