Why it matters
Metastore latency dominates query planning across many engines. A slow metastore slows Hive, Spark, Impala, and Presto simultaneously. Investing in metastore performance pays back across every analytical tool.
The metastore also drives governance. Ranger authorizes based on tables and columns as defined in the metastore. Atlas classifies them. Central schema management is what enables central governance.
The architecture
The metastore has two parts. The metastore service is a Java process exposing a Thrift API. It handles schema queries, partition operations, and stats lookups. Behind it is a relational database (MySQL, Postgres, Oracle) storing the actual metadata.
The schema in the database is straightforward: DBS holds databases, TBLS holds tables, PARTITIONS holds partitions with their locations, COLUMNS_V2 holds column definitions, TABLE_PARAMS and SERDES hold serialization info.
How it works end to end
Engines connect via Thrift (default port 9083) and issue calls like get_table, get_partitions, get_partitions_by_filter. These translate to SQL against the backend database.
Table partitions are the heaviest metadata. A partitioned table with a million daily partitions has a million rows in PARTITIONS. Listing them for a query without pruning is a large database operation.
Statistics tables (column stats, table stats) feed the CBO. Freshness matters: stale stats produce bad plans. Keep ANALYZE TABLE up to date.