Aiven Streaming Setup Guide

Kafka cluster, SSL certificates, and secrets for Module 8 (Optional)

title: “Aiven Streaming Setup Guide” subtitle: “Kafka cluster, SSL certificates, and secrets for Module 8 (Optional)” —

NoteFacilitator-managed setup

Your facilitator provisions the Aiven Kafka cluster, starts the sample data generator, runs the relay consumer, and loads Databricks secrets before Module 8. This page covers what you verify and run as an attendee.

Start on: Module 8 § Snowflake role · Exercise: Databricks § cluster · .env: Exercise: dbt § Configure .env · Module: 8 — Streaming (optional) · Lab: Exercise: Streaming

Overview

Module 8 uses Aiven Free Kafka as the streaming broker.

Component Who sets it up What it does
Aiven Kafka cluster Facilitator Apache Kafka broker (free tier)
User Activity generator Facilitator Publishes simulated events automatically
SSL certificates Facilitator Authenticates consumers to Kafka
Relay consumer (00_relay_consumer.py) Facilitator Writes Kafka events to ADLS2 as NDJSON
Databricks Secrets (workshop-scope) Facilitator SSL certs for streaming notebooks
Databricks Maven libraries You Kafka + Avro JARs on your cluster
Snowflake stage + Snowpipe You (SQL file) Loads NDJSON files into Bronze table
Snowflake Dynamic Tables You (SQL file) Silver + Gold streaming tables
Note.env (dbt stretch only)

Databricks notebooks and Snowsight SQL for Module 8 do not need .env. The optional dbt dynamic_table stretch reuses Exercise: dbt § Configure .env — verify with bash .devcontainer/setup-environment.sh before dbt run.

Trainee setup steps

1. Confirm Aiven credentials in Databricks Secrets

The Databricks streaming notebooks read SSL credentials from the workshop-scope secrets store. Your facilitator creates the scope and loads the values before Module 8 — see Exercise: Streaming prerequisites.

Verify (Databricks notebook or CLI) that these keys exist:

Secret key Value source
workshop-scope/aiven-bootstrap-servers Service URI from Aiven Console
workshop-scope/aiven-ca-cert Contents of ca.pem
workshop-scope/aiven-client-cert Contents of service.cert
workshop-scope/aiven-client-key Contents of service.key
workshop-scope/aiven-topic user-activity
Note

If any secret is missing, ask your facilitator — do not create workshop-scope yourself unless instructed.

2. Install Maven libraries on your Databricks cluster

The streaming notebooks require two Maven libraries that are not in the standard Databricks Runtime.

  1. Open your Databricks workspace → Compute → select your cluster
  2. Click Libraries tab → Install new
  3. Source: Maven — install both:
Coordinates Purpose
org.apache.spark:spark-sql-kafka-0-10_2.12:3.5.0 Kafka source/sink for Structured Streaming
org.apache.spark:spark-avro_2.12:3.5.0 Avro decoding (from_avro)
  1. Restart the cluster after installing

3. Set up Snowflake for streaming (Bronze + Snowpipe)

Requires Modules 2–3 setup (02_account_setup.sql + 04_external_stage.sql) so _workshop_config has attendee_id and sas_token.

  1. Open a Workspaces SQL file (Projects → Workspaces → + → SQL File)
  2. Open streaming/snowflake/01_setup_streaming.sql
  3. Run all as ACCOUNTADMIN — no placeholder editing; config auto-loads
  4. Switch to DE_WORKSHOP_ROLE and verify the Snowpipe was created (use your schema name):
SELECT SYSTEM$PIPE_STATUS(
    'DE_MASTERCLASS.DE_01_ALICE_STREAMING.STREAMING_PIPE_USER_ACTIVITY'
);  -- replace DE_01_ALICE with your attendee_id (uppercase)

4. Create Silver + Gold Dynamic Tables

After Bronze rows arrive via Snowpipe (relay consumer running):

  1. Open streaming/snowflake/02_dynamic_tables.sql in Snowsight
  2. Run all as ACCOUNTADMIN
  3. Wait ~1 minute, then run the verification queries at the bottom of the file
SHOW DYNAMIC TABLES LIKE 'STREAMING_%'
  IN SCHEMA DE_MASTERCLASS.DE_01_ALICE_STREAMING;  -- your {ATTENDEE_ID}_STREAMING

Verify Everything is Working

Databricks — confirm Kafka connection (Bronze notebook cell 1):

# Run first cell of 01_streaming_bronze.py
# Should print: "Kafka stream source ready"
# Should NOT raise: AuthenticationException or ssl.SSLError

Snowflake — confirm Bronze rows are landing:

-- Run this every 30 seconds after relay consumer starts
SELECT COUNT(*), MAX(ingest_ts) AS last_row
FROM DE_MASTERCLASS.{ATTENDEE_ID}_STREAMING.STREAMING_BRONZE_USER_ACTIVITY;

Expect first rows within 1–2 minutes of the relay consumer starting.

Troubleshooting
Issue Solution
ssl.SSLError: certificate verify failed Check that ca-cert secret contains the full content of ca.pem (no extra whitespace)
AuthenticationException Verify bootstrap-servers format is correct: hostname:port
Databricks: ClassNotFoundException: kafka Maven library not installed or cluster not restarted
Databricks: ClassNotFoundException: from_avro spark-avro library not installed
Bronze table empty after 3 min Confirm relay consumer is running; check ADLS2 container for files
Snowpipe not loading Run ALTER PIPE ... REFRESH; manually; check pipe status
Aiven service offline Free tier shuts down after 24h idle — restart from Aiven Console