Core concept
import org.apache.spark.sql.expressions.Window
val w = Window.partitionBy("dept").orderBy("salary".desc)
df.withColumn("rank", rank().over(w))Advertisement
How it works
Expensive for large partitions. Skew hurts.
Advertisement
Trade-offs + gotchas
Sessionization, top-N per group, running totals.