Why it matters

Bucketing is underused because Hive doesn't enforce it by default. Applications must write bucketed data correctly. When it works, sort-merge bucket joins on billion-row tables complete in seconds instead of hours.

Advertisement

The architecture

A bucketed table has CLUSTERED BY (cols) INTO N BUCKETS in its DDL. Each write must hash records by the clustered columns modulo N and route them to the corresponding bucket file.

Bucket file naming follows convention: 000000_0 through N-1_0. Additional file suffixes handle multiple write operations.

Bucketing layoutOriginal databillions of rowsHash functionhash % NBucket filesN files, balancedSort-merge bucket join on same-bucket-count tables avoids shuffle entirely
Rows hashed to fixed bucket files.
Advertisement

How it works end to end

To use bucketing effectively, both write and read sides must know about it. The writer must use INSERT with proper hashing (or use Hive's automatic hash routing when hive.enforce.bucketing is on).

Sort-merge bucket join optimization: when two tables are bucketed on the same key with the same count, their buckets align. Join reads bucket 0 of A with bucket 0 of B, bucket 1 with bucket 1, etc. No shuffle needed.

Statistical sampling: TABLESAMPLE(BUCKET x OUT OF y) picks a fraction of buckets for approximate queries.