Why it matters

Dedup common streaming need. Understanding shapes correctness.

Advertisement

The architecture

dropDuplicates([id column]).

State stores seen IDs bounded by watermark.

Streaming dedupdropDuplicateson event IDState stores IDsbounded by watermarkDuplicates droppedon matchWatermark essential to bound state — don't keep IDs forever
Dedup semantics.
Advertisement

How it works end to end

Watermark: bounds state size.

State size: proportional to unique IDs within window.

Alternative: sink-level idempotency.