Why architecture matters here
Spark + Iceberg fails on catalog misconfig, unmanaged snapshots, and small files. Architecture matters because paths + procedures + catalogs combine.
The architecture: every piece explained
The top strip is integration. Spark SQL / DF. Iceberg extension catalog + optimizer. Write path commit to log. Read path snapshot pruning.
The middle row is features. Procedures compact / migrate / rewrite. Catalyst push-down filter + projection. Streaming append. MERGE + CDC.
The lower rows are ops. Metrics + observability. Catalogs Hive/Glue/REST/Nessie. Ops — retention + snapshot expiry + concurrent writers.
End-to-end flow
End-to-end: streaming job appends events. MERGE upserts customer state. Nightly Spark job runs compact procedure. Read via Catalyst; filters pushed down; snapshot pruning skips old partitions.