Why architecture matters here
The MapReduce tax that Tez eliminates was structural, not incidental. Consider a query joining three tables with a group-by: as MapReduce, that is typically three-plus jobs, and between each, the full intermediate dataset is written to HDFS with replication, then read back. For a 100GB intermediate, that is hundreds of gigabytes of unnecessary IO per stage boundary, plus 10–30 seconds of job setup each time, plus cluster scheduling latency between jobs. The DAG model attacks the structure: one application, stage boundaries become edges with data flowing producer-to-consumer (pipelined where possible — a reducer can start merging while mappers still run), and intermediate data touches HDFS never. For ETL pipelines measured in hours, the same SQL routinely runs 3–10× faster on Tez purely from removed overhead.
The second structural matter is where parallelism decisions happen. A planner choosing reducer counts from table statistics guesses; skewed filters and stale stats make the guess wrong by 100× in either direction — thousand-task stages processing kilobytes, or ten-task stages choking on terabytes. Tez's runtime reconfiguration moves the decision to execution time: vertex managers observe actual output sizes from upstream tasks and set (or adjust) downstream parallelism accordingly — auto-reducer parallelism shrinking 1,009 planned reducers to 40 real ones is a routine log line on a healthy cluster. This is the difference between an engine that degrades gracefully with imperfect stats and one that requires perfect foresight.
Third: warm execution changes workload economics. Container reuse and session pools mean a cluster serving thousands of scheduled queries pays JVM startup and JIT warmup a handful of times per hour instead of per query — and it is why interactive Hive (even pre-LLAP) became plausible. Operationally, this converts tuning from per-job knobs into fleet management: container sizes, reuse policies, and AM pools are the levers, and misjudging them shows up as either wasted reservation or queue starvation.
The architecture: every piece explained
Top row: from SQL to a running DAG. HiveServer2 parses, optimizes (CBO with statistics, predicate pushdown, join reordering), and emits a physical plan as a Tez DAG description. The Tez application master — launched per query, or long-lived per session — turns the description into execution: it requests containers, schedules tasks with locality preferences, tracks attempts, and handles failures by re-running only affected tasks (a failed task re-executes; its consumed inputs re-fetch — not the whole query). Vertices are the DAG's nodes: each is a pipeline of Hive operators (scan → filter → project → partial aggregate) executed by N parallel tasks. Edges type the data movement: scatter-gather is the classic shuffle (each producer partitions to all consumers), broadcast replicates a small vertex's output to every consumer task (map-join distribution), one-to-one pipes task i to task i for stage fusion without redistribution.
Middle row: the execution substrate. Tasks run in YARN containers sized by configuration (the eternal hive.tez.container.size); container reuse keeps a finished task's JVM and hands it the next task — across vertices and, within a session, across queries — preserving JIT state and amortizing startup; idle containers release after a timeout so the queue reclaims capacity. The runtime graph machinery is the adaptivity: vertex managers receive events about upstream output sizes and can change parallelism (auto-reducer parallelism), swap edge implementations, or trigger partition pruning dynamically (a broadcast join's small side, once materialized, prunes the big side's splits). The shuffle layer moves edge data: sorted or unsorted, fetched from producer-local disk via the shuffle handler, merged in-memory when it fits, pipelined so consumers start before all producers finish.
Bottom rows: latency and quality-of-life. Sessions and AM pools: HiveServer2 maintains pre-warmed Tez sessions (AM + a set of containers) per queue; interactive queries grab one and skip AM startup — the pre-LLAP answer to BI latency, still the backbone for ETL orchestration. Vectorization and stats: operators process 1024-row batches (the same machinery LLAP uses), and statistics feed both the CBO and runtime decisions. The ops strip is real life: container size vs heap vs YARN limits, AM pool sizing per queue, and the Tez UI — per-vertex timelines, task counters, shuffle bytes — as the first stop for every 'why is this query slow'.
End-to-end flow
Trace a nightly ETL statement: INSERT into a fact table from a 2TB staging table joined to two dimensions, grouped and aggregated. HiveServer2's CBO orders the joins (dimensions broadcast; the big table streams), and emits a DAG: V1 scans dim_store (broadcast edge), V2 scans dim_product (broadcast edge), V3 scans staging and performs both map-joins plus partial aggregation (600 tasks from split calculation), V4 does final aggregation (planned parallelism: 1,009 — a stats-driven guess), V5 writes the output.
The session's AM (pre-warmed, from the ETL queue's pool) starts V1 and V2 immediately — they are small and finish in seconds, their outputs broadcast to containers as they come up. V3's 600 tasks flow through 180 reused containers in waves; each task's map-joins hash-probe in memory (the broadcast payloads arrived once per container, not once per task — reuse again), and partial aggregates spill sorted runs. As V3 tasks report output sizes, V4's vertex manager sums them: 38GB — auto-reducer parallelism reconfigures V4 from 1,009 to 76 tasks. Shuffle is pipelined: V4's tasks begin fetching and merging while late V3 stragglers still run. One V3 task dies with its node; the AM reschedules it elsewhere, V4's affected fetches retry, and the query proceeds — the failure cost is one task, not a restart. V5 writes partitions and the AM commits the DAG. Wall clock: 11 minutes; as MapReduce chains this plan historically ran 55.
The morning-after debugging story shows the other half. An analyst reports a slow variant of the query. The Tez UI's DAG timeline shows V4 with 4 tasks running long while 72 finished instantly — classic key skew (a null-heavy join key). The fix is in SQL and stats (null filtering, skew hints), diagnosed in five minutes because the DAG view exposes exactly which vertex, which tasks, and which counters (shuffle bytes per task) misbehaved. The operational moral: Tez turned both execution and diagnosis into per-vertex problems.