Why it matters

ORC drives some of the largest wins in Hive performance. Columnar + predicate pushdown + stripe skipping can turn a full-table scan into reading a few percent of the file.

Advertisement

The architecture

An ORC file is divided into stripes (default 64 MB). Each stripe contains column data plus indexes. Within a stripe, each column has its own contiguous byte range, dictionary-encoded and compressed.

The stripe footer lists column offsets and statistics. The file footer lists stripe offsets. The trailer at end of file points to the footer.

ORC file structureStripe (64 MB)column data + indexStripe footercolumn offsets + statsFile footer + trailerstripe list + pointsColumn pruning skips columns; predicate pushdown skips stripes via stats
ORC layout enables selective reads.
Advertisement

How it works end to end

Row group indexes within each stripe subdivide it into 10,000-row groups. Each group has min/max/count stats per column. Predicates like WHERE col > 100 can skip row groups whose max is under 100.

Bloom filters can be added for equality predicates. These sit alongside min/max and skip row groups where the target value provably isn't present.

Column pruning is automatic. Reading columns A and C from a file with columns A-Z reads only A and C data; the others are not fetched.