Why it matters
DataFrames enable Spark to compete with dedicated SQL engines. Catalyst optimization can turn a naive filter-then-join query into a plan that pushes the filter into the scan, uses a broadcast join, and vectorizes the aggregation. This is transparent — the user writes idiomatic code, Catalyst produces the optimized plan.
The architecture
A DataFrame is a Dataset of Row objects. Underneath, both use the Tungsten binary format: rows are packed into off-heap memory with column-oriented access. This bypasses JVM object overhead and enables SIMD-friendly processing.
Catalyst is the query optimizer. It represents the query as a tree of logical operators (Scan, Filter, Project, Join, Aggregate), applies rule-based rewrites (predicate pushdown, constant folding), and picks the cheapest physical plan.
How it works end to end
Datasets add typed operators. A Dataset[User] enforces at compile time that the fields exist and types match. This catches many bugs that Row-based DataFrames leave as runtime failures.
Encoder converts between JVM objects and Tungsten binary format. Kryo-based encoders handle arbitrary types; the built-in encoders are much faster for standard types.
Query execution: DataFrame is transformed by Catalyst, code-generated (Whole-Stage Codegen) into tight loops, and executed on Tungsten binary data.