Why architecture matters here

Kinesis architecture matters because it's opinionated on things Kafka leaves flexible. Shards are the throughput unit; consumers scale via enhanced fan-out; delivery via Firehose is one-click. The trade-off is less control.

Cost is per shard-hour + PUT payload. For steady moderate volume Kinesis is competitive; at very high volume MSK may be cheaper.

Reliability is strong; AWS handles replication + regional failover.

Advertisement

The architecture: every service explained

Walk the diagram top to bottom.

Producer. Uses PutRecord API or Kinesis Producer Library (KPL) for batching.

Kinesis Data Stream. The stream. Composed of shards; retention configurable.

Consumer. KCL (Kinesis Client Library) manages sharding + checkpointing. Or Lambda triggers.

Shards. Throughput unit: 1 MB/s write and 2 MB/s read + 1000 records/s write. Add shards to scale.

Retention. 24 hours default; up to 365 days paid.

Enhanced Fan-Out. Dedicated 2 MB/s per consumer; multiple consumers don't share throughput.

Firehose. Kinesis Data Firehose — one-click delivery to S3, Redshift, OpenSearch, HTTP endpoints. Handles batching + retry.

Managed Flink. Streaming SQL on Kinesis streams. Serverless-ish.

Auto-scale streams. Adjust shard count; keeps ordering within shard.

vs MSK Kafka. Managed simplicity vs Kafka flexibility.

ProducerPutRecord / KPLKinesis Data Streamshards + retentionConsumerKCL / LambdaShards1 MB/s + 1000 records/s per shardRetention24h default; up to 365 daysEnhanced Fan-Outdedicated throughput per consumerFirehoseS3 / Redshift / OpenSearch deliveryAnalyticsManaged FlinkAuto-scale streamsshard changesvs MSK Kafkamanaged vs self-managedAWS-managed streaming with Firehose delivery + Managed Flink analytics
AWS Kinesis architecture: Data Streams (shards, retention, EFO) + Firehose delivery + Managed Flink analytics; consumer models via KCL/Lambda.
Advertisement

End-to-end streaming flow

Trace a workflow. App emits events to Kinesis via KPL. KPL aggregates + compresses; sends PutRecords.

Stream has 10 shards. Partition key hashes to shard; ordered writes within shard.

Consumer: KCL library on 5 EC2 instances. Each takes 2 shards; checkpoints to DynamoDB.

Second consumer added later (analytics). Enhanced fan-out: gets its own dedicated 2 MB/s per shard; doesn't compete with first consumer.

Firehose subscribes: batches every 5 minutes or 5 MB; writes Parquet files to S3.

Managed Flink app runs SQL: "SELECT COUNT(*) FROM stream GROUP BY user_id TUMBLE 1 MINUTE." Results emit to another stream.

Volume spike; auto-scale adds 5 shards. Existing consumers continue; new shards distributed automatically.