Why architecture matters here
The economics of analytical storage are the economics of skipping. A 2B-row fact table answering a filtered query touches perhaps 0.1% of its rows; the difference between a 2-second and a 200-second query is whether the storage layer can prove the other 99.9% irrelevant without reading it. ORC's layered statistics are that proof system: file-level stats let the planner drop whole files (combined with partition pruning); stripe stats let the reader skip stripe reads; row-group stats — the workhorse — let it skip 10k-row windows inside stripes it must open; bloom filters extend skipping to point lookups where min/max ranges are useless (a needle ID in an unsorted haystack has min≤needle≤max everywhere). Every layer is metadata read before data, and the entire design only pays off when writers cooperate — which is the operational core of the format.
Write-time decisions determine read-time physics. Min/max skipping works when values cluster: a table sorted (or at least clustered) on its dominant filter column has tight, disjoint row-group ranges — the predicate isolates a few groups; the same data written unsorted has every group spanning the full value range — nothing skips. File sizing likewise: thousands of 10MB files mean footer-reading overhead dominates and stripes shrink below efficiency; a few 10GB files limit parallelism. Encoding choices (dictionary thresholds, string handling) shape both size and CPU. Teams that treat the writer configuration as an afterthought ship warehouses where the format's entire index machinery is present and useless.
And ORC's role has broadened: it is the physical substrate under Hive ACID (delete deltas address ORC's row identity; compaction rewrites are ORC merges), a peer of Parquet in the open-table-format world (Iceberg manifests point at ORC files whose stats feed Iceberg's own pruning), and the decode source for vectorized execution (streams map onto ColumnVectors with minimal transformation). Understanding the file is understanding the bottom layer of the whole stack's performance story.
The architecture: every piece explained
Top row: the container. The file layout ends with a postscript (compression kind, footer size) and a footer holding the schema (types as a tree — nested structs/lists/maps flatten into a numbered column hierarchy), stripe locations, and file-level column statistics: one small tail read makes the file navigable. Each stripe (default target 64MB+, often configured larger) is self-contained: an index section, a data section, and a stripe footer describing its streams. Column streams split each column into roles: PRESENT (null bitmap, RLE-compressed), DATA (values), LENGTH (for strings/lists), DICTIONARY_DATA (when dictionary-encoded); nested types decompose into streams per level. Encodings are chosen per column per stripe: RLEv2 for integers (with delta and patched-base modes for sequences and near-sorted data), dictionary vs direct for strings by cardinality ratio, and specialized paths for decimals and timestamps.
Middle row: the index machinery. Row groups (10,000 rows) are the skipping quantum: for each, the row index records per-column min/max/nulls and — the mechanical enabler — stream positions, so a reader can seek mid-stripe directly to group boundaries (positions include compression-block and RLE-run offsets; skipping is a seek, not a scan). Statistics stack: row-group entries roll up to stripe stats, stripe to file. Bloom filters, when enabled per column, sit alongside the row index (one filter per row group) sized by expected distinct counts and false-positive rate — the tool for WHERE id = ? on high-cardinality unsorted columns. Compression wraps every stream in independently-decompressible blocks (zlib for maximum ratio, zstd increasingly the default, snappy for speed), which is what allows positional seeking inside compressed data.
Bottom rows: the read contract and the mutable world. Predicate pushdown: engines hand the reader a SARG tree (col > 5 AND date BETWEEN …); evaluation runs coarse-to-fine — file stats, stripe stats, row-group stats, bloom filters — producing the set of row groups to actually decode; everything else is skipped by seeking. Decoded streams fill vectorized batches (isRepeating from RLE runs, dictionary indices resolved or kept for late materialization). ACID and schema evolution: transactional tables embed the (originalTransaction, bucket, rowId) identity in ORC's structure so delete deltas can address rows; schema evolution follows column-ID matching — adds are safe (missing columns read as null), type widenings are supported, renames depend on ID-based (not name-based) mapping configuration — the rules that decide whether old files survive new schemas. The ops strip: file sizing to stripe multiples, sort-on-write for the dominant predicate, and verifying (not assuming) that stats and blooms exist where queries need them — orcfiledump is the truth tool.
End-to-end flow
Follow one file's life. A nightly ETL writes a day of transactions: the writer is configured with 128MB stripes, zstd, bloom filters on transaction_id, and — the decision that matters most — rows sorted by merchant_id within the partition (the dominant analytical filter). As rows stream in, the writer buffers a stripe's worth, chooses encodings per column (merchant_id: RLEv2 with long runs thanks to the sort; status: dictionary with 6 entries; amount: direct), records row-index entries every 10k rows with positions and min/max, builds blooms, compresses streams, and flushes the stripe. Eight stripes and one footer later, the file is 1.1GB of self-indexed columnar data.
Morning queries exercise the layers. WHERE merchant_id = 88412 AND dt = yesterday: partition pruning picks the file; file stats say merchant range covers 88412; stripe stats eliminate six of eight stripes; within the two survivors, row-group min/max — tight, because sorted — isolate 5 of 190 groups; the reader seeks to their positions, decodes three columns' streams into vector batches, and the query touches 50k rows of 14M. The point lookup WHERE transaction_id = 'TX-9F3A…' would find min/max useless (IDs unsorted, every group spans the space) — the bloom filters answer instead: 189 groups say 'definitely not', one says 'maybe', one group decodes, one row returns. The un-blessed query — a filter on a column nobody sorted or bloomed — scans everything, and the profile's 'row groups read: all' line is the signature to recognize in triage.
The file then lives through the table's evolution. A month later, ALTER TABLE adds channel: readers of the old file materialize nulls for the missing column ID — no rewrite. A GDPR deletion lands via Hive ACID: a delete delta addresses rows in this file by identity triple; readers merge-subtract until major compaction physically rewrites the survivors into a fresh, still-sorted ORC file. And at quarter's end, the platform's file-health job flags the table's other pipeline — a streaming ingest producing 8MB files with 4MB stripes and no sort — as the source of the 'why are merchant queries slow on recent data' ticket: same schema, same format, none of the physics. The fix is a compaction step with the blessed writer config; the lesson is the format's recurring one — ORC gives exactly the performance the writer paid for.