Why it matters
Every Hive performance and reliability issue can be traced to one of these components. HiveServer2 crashes take down every client. Metastore latency slows every query. Compiler bugs produce wrong plans. Knowing which component owns which failure mode is what makes operations fast.
The architecture
HiveServer2 is a Thrift server that accepts JDBC and ODBC connections. It manages sessions, authentication (Kerberos or LDAP), and per-session state. Each session has a driver instance.
The driver receives SQL, calls the compiler, submits the plan to the execution engine, and returns results. It also handles session context: current database, session variables, and open transactions in ACID mode.
The compiler runs stages: parse (SQL → AST), semantic analysis (resolve tables/columns), logical plan generation, logical plan optimization (predicate pushdown, join reordering), physical plan generation, and physical plan optimization (map-side aggregation, bucketed joins).
How it works end to end
During compilation, the compiler consults the metastore multiple times: table schemas, partition lists, statistics for the cost-based optimizer, and index locations. Metastore performance directly affects compile time.
After compilation, the plan is submitted to the execution engine (MR, Tez, or Spark). The engine reads HDFS files based on the plan's split definitions, executes operators, and writes intermediate or final output. Progress is reported back via job trackers.
Results either return through the driver to the client (for small results) or are written to a scratch directory and streamed back.