Why architecture matters here
Consider the alternative. If every coordinator resolved metadata at plan time, a query over a table with 100k partitions would begin with 100k-entry metastore calls and object-store listings — tens of seconds before a single scanner thread started, multiplied across every concurrent query, hammering a metastore that was never built for OLAP fan-out. Engines that plan against the metastore directly pay this on every cold query; Impala pays it once, in the background, in catalogd.
Centralizing writes in one daemon buys three properties. Coherence with cheap reasoning: because only catalogd mutates metadata, a single monotonically increasing version number totally orders all changes; a coordinator is simply ‘caught up to version V’, and the system can express guarantees like SYNC_DDL (block until every coordinator has V) in one integer comparison. Metastore protection: the Hive Metastore and NameNode see one polite client instead of a coordinator mob; a query storm cannot become a metastore outage. Plan-time determinism: planning latency is a function of cache lookups, not external-system weather, which is what keeps short queries short.
The costs are equally structural. catalogd’s JVM heap must hold the working set of table, partition, and file metadata — on wide clusters this reaches tens of gigabytes, and a full reload after restart can take many minutes. Broadcast fan-out means every coordinator holds a copy, multiplying the RAM bill. And staleness is now a user-visible concept with explicit repair verbs. The local-catalog (on-demand) mode introduced to fix the RAM problem — coordinators fetch metadata granularly from catalogd and cache with eviction — is precisely a renegotiation of this trade: less RAM, more plan-time RPCs, same single-writer core.
The contrast with per-query metadata engines sharpens the point. Trino resolves metadata through connector calls at planning time, softened by short-TTL caches — a fine fit for federated, exploratory work where freshness beats plan latency. Impala’s bet is the opposite because its target workload is the opposite: thousands of repetitive BI dashboard queries per hour against a stable set of curated tables, where a 50ms plan against week-old file lists that are refreshed on ingest beats a 2-second plan against live listings every time. The statestore also carries more than the catalog: cluster membership (which executors are alive, feeding the scheduler) and admission-control state (running queries and queued memory per pool) ride the same pub/sub plane. That makes statestore health a query-plane concern too — stale membership means scan ranges assigned to dead executors and queries failing at runtime rather than queueing politely.
The architecture: every piece explained
catalogd. The metadata authority. On startup it enumerates databases and table names from the Hive Metastore but loads table detail lazily — a table’s partitions, file descriptors, and block locations are pulled on first reference (or eagerly if --load_catalog_in_background is set). Each loaded object carries a catalog version. DDL executed through Impala goes coordinator → catalogd, which writes to the metastore, updates its own cache, bumps the version, and returns the new objects to the requesting coordinator inline so the issuing session sees its own writes immediately.
statestored. A membership and topic broker. Subscribers register topics (catalog-update, cluster membership, admission-control state) and receive periodic delta updates plus heartbeats. It keeps no durable state: on restart, subscribers re-register and topics rebuild. It is intentionally not a consensus system — it is a fan-out optimization, and its failure degrades freshness, not correctness.
Coordinator caches. Each impalad that accepts queries applies catalog deltas to a local cache keyed by version. In legacy mode the cache is complete (everything catalogd has); in local-catalog mode (--catalog_topic_mode=minimal) the topic carries only invalidations and coordinators fetch object detail on demand with LRU eviction — the fleet RAM bill drops from O(coordinators × catalog) to O(working set).
The repair verbs. REFRESH t reloads file and block metadata for a known table (cheap, incremental per partition); INVALIDATE METADATA t drops the object so the next reference reloads everything (expensive); bare INVALIDATE METADATA drops the world — the nuclear option that makes the next minutes of queries pay cold-load costs. Event polling (--hms_event_polling_interval_s) subscribes catalogd to metastore notification events so external Hive/Spark writes propagate automatically, shrinking the need for manual repair.
Deployment topology completes the picture. Only daemons started as coordinators (--is_coordinator) subscribe to the catalog topic and carry the cache; dedicated executors (--is_executor) hold none of it, which is why the standard large-cluster pattern — a handful of dedicated coordinators fronted by a load balancer, dozens of executor-only nodes — is also a metadata-plane optimization: catalog fan-out cost scales with coordinator count, not cluster size. For availability, recent releases support catalogd and statestored HA pairs: an active/standby catalogd elected via the statestore, with coordinators failing over their RPC target automatically. HA converts the historical single-point-of-slowness into a restartable role — though the standby’s cache still warms lazily, so failover restores service faster than it restores latency.
End-to-end flow
Trace ALTER TABLE sales ADD PARTITION (day=‘2026-07-11’) issued at Coordinator 2 with SYNC_DDL=1. The coordinator does not touch the metastore; it sends the DDL to catalogd. catalogd writes the partition to the Hive Metastore, lists the new partition directory on HDFS/S3 to build file descriptors, updates its in-memory object, and stamps the change with version, say, 4213. The response returns the updated table inline, so Coordinator 2 applies it immediately — read-your-writes for the issuing session. catalogd’s next delta to the statestore topic carries the versioned change; statestored fans it to Coordinators 1 and 3 on the next update cycle (typically sub-second). Because SYNC_DDL is set, Coordinator 2 holds the DDL’s completion until the statestore confirms all live coordinators have acknowledged a topic version ≥ 4213 — the client’s next query is correct from any coordinator behind the load balancer.
Now trace a cold SELECT on a table Impala has never touched in legacy mode. Coordinator 1’s cache has only the table name (an IncompleteTable stub). Planning triggers a prioritized load request to catalogd, which pulls the table from the metastore, lists every partition directory for file metadata, fetches block locations, assembles what can be hundreds of megabytes of thrift objects, bumps the version, and ships the table back (and to the topic). The query then plans locally: partition pruning against cached partition keys, scan-range construction against cached block locations, no external calls. The second query on the table skips all of it — that is the architecture working. In local-catalog mode the same flow fetches only the referenced partitions’ metadata, trading fleet RAM for occasional re-fetches after eviction.
One more flow worth tracing is COMPUTE STATS sales, because it exercises every layer at once. The coordinator runs the child queries (row counts, per-column NDVs), then hands the results to catalogd as a DDL-like update; catalogd writes the stats into the metastore, updates its cached table, bumps the version, and broadcasts. Within an update cycle, every coordinator’s planner sees the new NDVs and starts choosing better join orders — a cluster-wide planner behavior change propagated by the same versioned-delta machinery as a schema change, with no restart and no cache flush. It is a useful mental model: anything that should change planning everywhere — stats, a new partition, a permission — must ride the catalog topic, and its propagation latency is the topic’s update interval, observable and boundable.