Why architecture matters here
Data lake architecture matters because "put everything in S3 and query it later" collapses under real workloads. Without schema management, queries can't find data. Without access control, sensitive data leaks. Without partition pruning, queries scan terabytes. The architecture is what makes the lake usable.
Cost is a huge lever. Parquet compression + partition pruning + Athena's per-query pricing can be dramatically cheaper than a data warehouse. But careless queries scan the whole lake and burn money.
Reliability is where Lake Formation + Iceberg/Delta shine. Access controls persist across services; table formats give ACID and time travel.
The architecture: every layer explained
Walk the diagram top to bottom.
Ingest. Kinesis for streams; AWS DMS for DB CDC; DataSync for on-prem; Kafka + connectors; scheduled batch imports.
S3 Raw Zone. Data as ingested. JSON/CSV/logs/Parquet with minimal transformation. Retention long (durable archive).
S3 Curated Zone. Cleaned, normalized, Parquet-formatted, partitioned. Iceberg or Delta table format for ACID.
Glue Catalog. Central metastore. Tables, schemas, partitions, columns. Consumed by Athena, EMR, Redshift, Databricks.
Lake Formation. Fine-grained access control on catalog resources. Row + column filters. Applies across services querying via Glue Catalog.
ETL. Glue for serverless Spark; EMR for provisioned Spark/Presto. dbt runs on top for transformation frameworks.
Query. Athena for serverless SQL — pay per scanned data. Presto/Trino on EMR for high-throughput.
BI. QuickSight (AWS native), Tableau, Looker — connect via Glue Catalog + Athena.
Observability. CloudWatch for AWS-native services; Datadog for cross-cloud/tool.
Cost + Governance. Tag resources by team; Lake Formation tag-based access; S3 storage class transitions to cut cost.
End-to-end data flow
Trace a workflow. Application emits events to Kinesis. Firehose writes them to S3 raw zone as JSON with 5-minute batches, partitioned by date.
Nightly Glue job reads raw, deduplicates, normalizes schema, writes curated Parquet to Iceberg table partitioned by event_type + date.
Glue Crawler updates catalog with new partitions.
Lake Formation applies access policy: analytics team can query event_type IN ('purchase', 'signup') but not other events; PII columns masked for their role.
Analytics team queries via Athena: SELECT day, COUNT(*) FROM curated.events WHERE event_type='purchase' AND day BETWEEN '2026-05-01' AND '2026-05-31' GROUP BY day. Athena scans only the purchase partition + date range; scans 200 MB not 200 TB. Cost: cents.
Dashboard in QuickSight connects to Athena. Analysts see the results in seconds.
Data science team requests time-travel access to compare a metric today vs last month. Iceberg's time-travel feature: SELECT ... FROM events VERSION AS OF '2026-05-01'. Same query returns as-of-then data.
Auditor requests: who accessed what PII last quarter? Lake Formation access logs answer.