Why it matters

Spark SQL made Spark accessible to analysts who don't want to write DataFrame code. It's also faster than Hive for most workloads and more portable than Impala. This has driven a lot of Hive workload migration to Spark SQL over the past several years.

Advertisement

The architecture

The SQL query goes through the same Catalyst optimizer as DataFrames. Parser (SQL → unresolved plan), analyzer (resolve tables/columns via metastore), optimizer (rule + cost based), and physical planner produce the final DAG.

Execution uses Tungsten and whole-stage codegen. The result is a plan that reads columnar files, applies filters via vectorized code, joins with efficient hash or sort-merge, and aggregates with hash-based accumulators.

Spark SQL pipelineParse + analyzeSQL → planOptimizeCatalystExecuteTungsten codegenSame Catalyst engine as DataFrames; SQL is just another entry point
SQL through Catalyst.
Advertisement

How it works end to end

Metastore integration: Spark SQL can use the Hive Metastore. This gives compatibility with Hive tables and cross-engine table sharing.

Thrift server exposes Spark SQL via JDBC/ODBC. BI tools connect to it as a SQL warehouse.

Adaptive Query Execution (AQE) reoptimizes plans at runtime based on actual data statistics. This handles skew, coalesces small partitions, and switches join strategies mid-query.