MHP

Streaming Patterns — Kafka to Medallion

Real-time data engineering with Databricks Structured Streaming & Snowflake Dynamic Tables
Kafka Fundamentals
Topics, partitions & offsets
Consumer groups for parallelism
Avro + Schema Registry (Karapace)
Structured Streaming
readStream / writeStream API
Watermarks for late-arriving data
Checkpoints for exactly-once semantics
Streaming Medallion
Bronze: raw append from Kafka
Silver: filter, enrich, deduplicate
Gold: sliding windows & aggregations

Batch vs Streaming — most use cases don’t need streaming; use it when latency genuinely matters (sub-minute action required)