Why architecture matters here
Data lake architecture matters because "put everything in S3 and query it later" collapses under real workloads. Without schema management, queries can't find data. Without access control, sensitive data leaks. Without partition pruning, queries scan terabytes. The architecture is what makes the lake usable.
Cost is a huge lever. Parquet compression + partition pruning + Athena's per-query pricing can be dramatically cheaper than a data warehouse. But careless queries scan the whole lake and burn money.
Reliability is where Lake Formation + Iceberg/Delta shine. Access controls persist across services; table formats give ACID and time travel.
S3 is a key-value store wearing a filesystem costume
S3 is a flat key-value namespace: no directories, only object keys containing slashes. The console renders raw/events/dt=2026-08-01/part-0000.parquet as a folder tree; the service stores one key. Three consequences follow.
Listing is an API call, not a directory read. ListObjectsV2 returns at most 1,000 keys per request, so a table spread across a million small objects costs a thousand round trips of metadata before an engine reads a byte of data — a lake that feels slow at trivial scan volume is spending its time in planning.
Request rate scales per prefix. S3 sustains at least 3,500 write and 5,500 read requests per second per partitioned prefix and splits prefixes automatically as traffic grows, so writes spread across many high-order prefixes absorb bursts that one hot prefix would throttle.
Prefix layout is policy, not aesthetics. A boundary is where you hang a bucket policy, an IAM resource path, a lifecycle rule, a replication rule, and an Inventory report.
# one bucket per zone: separate lifecycle and policy
s3://acme-lake-raw/ source=stripe/ingest_dt=2026-08-01/part-0000.json.gz
s3://acme-lake-bronze/ db=payments/table=charges/dt=2026-08-01/part-0000.parquet
s3://acme-lake-silver/ db=payments/table=charges_conformed/...
s3://acme-lake-gold/ db=finance/table=daily_revenue/...The architecture: every layer explained
Walk the diagram top to bottom. Ingest lands data in S3 through Kinesis and Firehose for streams, DMS for database change capture, DataSync for on-premises stores, or scheduled batch imports. The raw and curated zones live on S3; the Glue Data Catalog describes them; Lake Formation governs who sees what; Glue and EMR transform; Athena, Redshift Spectrum, and EMR query; QuickSight and Tableau visualise. The sections that follow take each layer in turn.
Zones: raw, bronze, silver, gold
Raw holds bytes exactly as they arrived, in the source's own encoding — gzipped JSON, CSV with its broken quoting, a DMS change stream — and is append-only. Bronze is the same data mechanically converted to partitioned Parquet, no business semantics applied. Silver is conformed: deduplicated, type-corrected, late arrivals reconciled, reference joins done once instead of in every downstream query. Gold is aggregates and marts shaped for one consumer.
How many zones you have matters far less than the invariants you hold. Raw is never written by a transformation job, only by ingestion, and only expired by a lifecycle rule. Every other zone must be reproducible by replaying from raw, which turns a logic bug from an outage into a backfill. Each zone gets its own bucket, or at least its own IAM boundary, so a defect in a gold job cannot corrupt the replay source.
The Glue Data Catalog is the contract
The catalog is what makes a lake a lake rather than a folder of files: a Hive-metastore-compatible store of tables, column types, serde, partition list, and S3 location. Athena, EMR Spark and Trino, Redshift Spectrum, and Glue jobs read the same entries, so one table serves five engines without five copies of the schema.
Crawlers are an exploration tool that keeps getting deployed as production infrastructure. A crawler samples files, infers types, and registers discovered partitions — excellent for exploration, a liability in production. Crawl a day on which a numeric field arrived quoted and the column widens to string, after which every downstream aggregate silently changes behaviour.
Production tables should be declared with explicit DDL in version control, and partitions registered by the job that wrote them — ALTER TABLE ... ADD PARTITION after a successful write, so the catalog only advertises data that landed completely. Athena's partition projection goes further for time-shaped tables: declare the key's type, range, and path format as table properties and the engine computes partition locations arithmetically instead of listing tens of thousands of partition rows.
Open table formats: what Iceberg adds over bare Parquet
Parquet under Hive-style directories gives columnar compression and partition pruning and nothing else: no atomic commit, so a reader listing a prefix mid-write sees a half-finished partition; no row-level update or delete, so correcting one record means rewriting a whole partition.
Iceberg replaces the directory listing with a metadata chain: a current-metadata pointer refers to a snapshot, which refers to a manifest list, which refers to manifests enumerating data files with per-file column min/max statistics. A commit is an atomic swap of that pointer, so readers only see complete snapshots, and because file statistics live in metadata the engine prunes files without listing S3 at all. On top of that: hidden partitioning (an analyst filters a timestamp, the engine derives the day partition), partition evolution without rewriting history, schema evolution by field ID, snapshot time travel, and row-level MERGE/DELETE in copy-on-write or merge-on-read form.
Delta Lake reaches similar guarantees through a JSON transaction log with Parquet checkpoints and suits Spark-dominated stacks; Hudi is built around record-level indexes for upsert-heavy CDC. Iceberg has the widest first-party reach on AWS. The price of all three is identical: metadata is state you own, so snapshots must be expired, orphan files removed, and small files compacted.
End-to-end data flow
Trace a workflow. Application emits events to Kinesis. Firehose writes them to S3 raw zone as JSON with 5-minute batches, partitioned by date.
Nightly Glue job reads raw, deduplicates, normalizes schema, writes curated Parquet to Iceberg table partitioned by event_type + date.
Glue Crawler updates catalog with new partitions.
Lake Formation applies access policy: analytics team can query event_type IN ('purchase', 'signup') but not other events; PII columns masked for their role.
Analytics team queries via Athena: SELECT day, COUNT(*) FROM curated.events WHERE event_type='purchase' AND day BETWEEN '2026-05-01' AND '2026-05-31' GROUP BY day. Athena scans only the purchase partition + date range; scans 200 MB not 200 TB. Cost: cents.
Dashboard in QuickSight connects to Athena. Analysts see the results in seconds.
Data science team requests time-travel access to compare a metric today vs last month. Iceberg's time-travel feature: SELECT ... FROM events VERSION AS OF '2026-05-01'. Same query returns as-of-then data.
Auditor requests: who accessed what PII last quarter? Lake Formation access logs answer.
Picking a query engine
Athena is serverless Trino with nothing to provision. It bills per terabyte scanned, rounded up to a 10 MB minimum per query, which makes it nearly free for well-partitioned dashboard queries and brutally expensive for an unfiltered SELECT * over a fact table. It is the right default for ad-hoc analysis and any workload whose duty cycle would leave a cluster idle.
Redshift Spectrum exposes catalog tables as external tables inside a Redshift cluster or serverless workgroup. Its niche: the gold layer already lives in Redshift and you want to join warehouse dimensions to lake-resident facts without loading them.
EMR gives you provisioned Spark, Trino, or Hive with control over runtime version, libraries, memory, and instance mix including Spot. It wins for heavy transformation, ML feature pipelines, custom code rather than SQL, and any workload utilised enough that per-hour compute beats per-terabyte scanning. Serverless Spark on Glue sits between the two; its job model, DPUs, and bookmarks are covered in AWS Glue serverless Spark ETL, and the BI layer in Amazon QuickSight.
Partitioning and file-size economics
Two knobs account for most of what a lake costs: partitioning decides how much data the engine may skip, file size decides how efficiently it reads what is left. Partition on the column that appears in nearly every predicate — in practice, time — at the coarsest granularity that still prunes usefully. Every extra key multiplies the count: date by event type by region over two years is 730 x 20 x 15 = 219,000 partitions, and if each holds a few megabytes you have not saved scan cost, you have converted it into planning cost.
Aim for files between roughly 128 MB and 1 GB. Below about 64 MB the fixed cost per file — a GET, a Parquet footer read, a split assignment — dominates the useful work, and a table of 10 KB files is a request-charge generator. Parquet's row group, commonly 128 MB in Spark writers, is also the unit at which min/max statistics enable skipping, so files smaller than a row group forfeit most predicate pushdown too.
-- Iceberg housekeeping, runnable from Athena
OPTIMIZE payments.charges REWRITE DATA USING BIN_PACK
WHERE dt >= DATE '2026-08-01';
-- drop expired snapshots and unreferenced files
VACUUM payments.charges;Governance, encryption, and tiering
Plain IAM secures a lake at prefix granularity, which is too coarse the moment one table mixes columns different teams may see. Lake Formation sits above the catalog and becomes the permission authority for locations registered with it; engines going through the catalog receive temporary vended credentials scoped to the grant instead of using the caller's own S3 permissions. That indirection is what makes column masking and row filters enforceable rather than advisory — the engine never holds credentials broad enough to read the file and ignore the policy.
LF-Tags label databases, tables, and columns, and grants are written against tag expressions, so a new table inherits policy at creation. Data cell filters combine a row predicate with a column include/exclude list per principal — how one physical table serves an analytics team that must not see PII and a fraud team that must.
Encrypt at rest with SSE-KMS using a separate customer-managed key per zone; the full comparison of modes is in S3 encryption options. The lake-specific wrinkle is that KMS bills per request, so a scan touching 200,000 objects issues 200,000 decrypt calls. S3 Bucket Keys move data-key derivation to a bucket-level key and cut that traffic sharply — AWS documents up to 99% fewer KMS requests. Enforce aws:SecureTransport in the bucket policy and route engine traffic over a gateway VPC endpoint, so lake reads never leave the AWS network.
Tiering follows the zone model: raw moves from Standard through Standard-IA to Glacier Flexible Retrieval or Deep Archive as reprocessing becomes implausible. Mind the minimum billable durations — 30 days for IA, 90 for Glacier Flexible, 180 for Deep Archive — and the per-object transition charge, which can make moving a million tiny files cost more than it saves. Compact first, then tier.
Failure modes you will actually hit
Small files. Streaming ingest with a short flush interval produces thousands of objects per hour; planning slows and reading metadata ends up costing more than reading data. Fix with scheduled compaction and a larger ingest buffer.
Partition explosion. Someone adds a high-cardinality partition key — user ID is the classic — and the table acquires millions of partitions holding kilobytes each. Such columns belong in sort order or a bucketing scheme, never in the partition path.
Schema drift. An upstream service renames a field or changes a type and, because a crawler re-infers the schema, the change propagates silently. Check the contract at ingest instead of discovering it three dashboards later.
Unbounded scan cost. One SELECT * without a partition filter over a petabyte-scale table can outspend the rest of the month. Set per-query and per-workgroup data-scanned limits in Athena, and alarm on scanned bytes.
The raw zone becomes a landfill. Nothing is cataloged, nobody knows the owner, and years later nothing can safely be deleted. Make a catalog entry, an owner tag, and a retention rule the price of onboarding a feed.
When a lake is the wrong tool
A lake earns its operational surface when data is large, arrives in several formats, is read by more than one engine, and has consumers you cannot enumerate at write time. Remove any of those and something simpler wins: Redshift or Aurora alone for SQL over well-known tables at warehouse-friendly volume; DynamoDB for point lookups by key, since no partitioning scheme makes S3 competitive on single-row latency.