Why it matters
Compaction lag is the top ACID operational pain point. Falling behind causes slow reads; too-aggressive compaction wastes I/O. Getting the balance right takes tuning.
The architecture
Two compaction types: minor merges several small delta files into one larger delta; major merges base + all deltas into a new base, applying all mutations. Both run as YARN jobs.
A compactor thread on HiveServer2 monitors table states and schedules compactions when thresholds are exceeded. Deltas-per-partition threshold triggers minor; delta-to-base ratio triggers major.
How it works end to end
Minor compaction is cheap: reads a few small deltas, writes one bigger delta. Doesn't need to touch base file. Should run frequently.
Major compaction is expensive: reads base file + all deltas, applies all mutations, writes new base. Should run periodically but not too often.
Compaction is per partition (in partitioned tables). Each partition can have its own compaction schedule.