Why it matters

The move from MR to Tez was one of the largest single wins in Hive's history. Queries that took minutes started taking tens of seconds. Interactive query engines (Impala, Presto) became less compelling because Tez closed most of the gap.

Advertisement

The architecture

A Tez DAG has vertices (compute stages) connected by edges (data flow). Each vertex has multiple tasks running in parallel across containers. Edges have properties: broadcast (small side of a broadcast join), scatter-gather (shuffle), or one-to-one (map-only pass-through).

The Tez Application Master runs one AM per session or query. It manages the DAG, requests containers from YARN, and coordinates task execution.

Tez DAG for a group-by queryVertex 1: scanreads HDFSVertex 2: shuffle+agggroups by keyVertex 3: outputwrites HDFSData flows vertex-to-vertex without hitting HDFS in between; huge speedup vs MR chain
Tez DAG vs MR job chain.
Advertisement

How it works end to end

Container reuse is the biggest optimization. In classical MR, each stage got fresh containers, paying JVM startup cost each time. In Tez, containers are reused across stages within a query and even across queries in the same session. JVM startup cost is paid once per session.

Session mode keeps AMs and containers warm across multiple queries. First query pays startup cost; subsequent queries are sub-second on the same session.

The DAG optimizer rewrites plans to exploit Tez features: broadcast small tables to eliminate shuffle, use one-to-one edges for map-only stages, choose bucketed joins when possible.