Core concept

val rdd = sc.parallelize(1 to 1000, 4).map(_ * 2)
Advertisement

How it works

No schema. No Catalyst optimization. Verbose Java/Scala.

Advertisement

Trade-offs + gotchas

Custom partitioners. Non-tabular data.