Core concept

.writeStream
  .option("checkpointLocation", "/checkpoints/mystream")
  .start()
Advertisement

How it works

Checkpoint location must be reliable. Cannot change output schema after checkpoint.

Advertisement

Trade-offs + gotchas

Every streaming job.