Why it matters
Most Hive performance problems are diagnosable only by understanding execution. A slow query might be spending time in compilation, in DAG generation, or in actual data processing. Each has different fixes.
The architecture
Parse: SQL is tokenized and parsed into an abstract syntax tree by ANTLR-generated grammar rules.
Semantic analysis: table and column references are resolved against the metastore. Type checking happens. Complex expressions like windowing functions are validated.
Logical plan: the AST is converted to a tree of relational operators (scan, filter, project, join, aggregate). This is the algebraic representation.
How it works end to end
Optimization: predicate pushdown moves filters closer to scans. Join reordering picks the cheapest join order based on statistics. Column pruning eliminates unneeded columns. Partition pruning restricts scans to matching partitions.
Physical plan generation converts logical operators to physical operators (map-side vs shuffle-side aggregation, map join vs shuffle join). The result is a DAG of stages.
Execution submits the DAG to the engine. For Tez, the DAG becomes a Tez DAG with vertices and edges. For Spark, stages. For MR, sequential MR jobs. The engine runs the work, reads HDFS, writes intermediate output, and eventually returns results.