Why architecture matters here

Streaming failures come from checkpoint loss + wrong sink semantics. Architecture matters because Kafka + checkpoint + sink compose.

Advertisement

The architecture: every piece explained

The top strip is source. Kafka topic. Kafka source. Micro-batch. Offset tracking.

The middle row is delivery. Checkpoint dir. Sink. Exactly-once. Rate limit + max offsets.

The lower rows are ops. Schema evolution. Metrics. Ops — restart + rescale + retention.

Structured streaming + Kafka — offsets + checkpoint + exactly-onceSpark reads Kafka with correct semanticsKafka topicpartitionedKafka sourcereadStreamMicro-batchtriggerOffset trackingstart / endCheckpoint dirdurable stateSinkDelta / Kafka / filesExactly-oncecommit atomicRate limit + max offsetsbackpressureSchema evolutionAvro / JSON driftMetricslag + input rateOps — restart + rescale + retentiondurablewriteatomicthrottleevolvewatchwatchoperateoperate
Spark structured streaming + Kafka source with checkpoint.
Advertisement

End-to-end flow

End-to-end: readStream from Kafka. Trigger every 30s pulls up to 100k offsets. Writes to Delta table with checkpoint dir. Exactly-once achieved via Delta's transactional writes. Schema versioned via Avro registry.