Machine Learning Setup Guide

Databricks ML Runtime, Snowflake Cortex, and Snowpark ML for Module 9 (Optional)

title: “Machine Learning Setup Guide” subtitle: “Databricks ML Runtime, Snowflake Cortex, and Snowpark ML for Module 9 (Optional)” —

NoteOptional Module

This setup is only needed if you plan to run Module 9 (Machine Learning). Modules 2 and 3 (Databricks and Snowflake batch pipelines) must be completed first — the ML module reads from the Silver enriched tables produced by those pipelines.

Start on: Prerequisites · Module 6 § Cortex access · Codespace .env: Exercise: dbt § Configure .env · Module: 9 — ML (optional) · Lab: Exercise: ML

Prerequisites

Before starting Module 9:

  • silver_nyc_taxi_enriched exists in Databricks ({attendee_id}_silver schema)
  • SILVER_NYC_TAXI_ENRICHED exists in Snowflake ({ATTENDEE_ID}_SQL_SILVER — recommended SQL track)
  • Gold KPI pipeline complete in Snowflake ({ATTENDEE_ID}_SQL_GOLD for Cortex ML.FORECAST Step A0)

If any of these are missing, run the Bronze → Silver → Gold notebooks from Modules 2 and 3 first.

Note.env (Codespace tracks only)

Cortex SQL and Databricks sklearn do not need .env. For dbt ml_features_tip_prediction or Snowpark ML (02_snowpark_ml_tip_prediction.py) in the Codespace terminal, complete Exercise: dbt § Configure .env (Module 4) and verify:

bash .devcontainer/setup-environment.sh

Databricks Setup

Runtime requirement

The ML notebooks require Databricks Runtime ML (not standard Runtime).

How to check your cluster runtime:

  1. Compute → select your cluster → Edit
  2. Under Databricks Runtime Version, look for a version ending with ML — e.g., 16.4 LTS ML (recommended) or 15.4 LTS ML (minimum)
  3. If you have standard Runtime, create a new cluster with an ML runtime
TipWhat’s in the ML Runtime?

Databricks Runtime ML pre-installs: scikit-learn, XGBoost, LightGBM, PyTorch, TensorFlow, MLflow. You don’t need to pip install any of these.

Verify scikit-learn and MLflow are available

In a notebook cell:

import sklearn
import mlflow
import xgboost

print(f"sklearn: {sklearn.__version__}")
print(f"mlflow:  {mlflow.__version__}")
print(f"xgboost: {xgboost.__version__}")

Expected: all three import without error.

MLflow Experiment access

The ML notebook logs runs to MLflow Experiments (built in to Databricks).

  1. Click Experiments in the left sidebar
  2. Verify you can create an experiment (or that a shared one exists)
  3. After training, your run appears here with metrics, params, and model artifacts

Fallback: standard Runtime

If only standard Runtime is available, install the required packages at the top of the notebook:

%pip install scikit-learn xgboost mlflow pandas matplotlib

Snowflake Setup

1. Cortex ML access (self-service on trial)

Cortex ML Functions require the SNOWFLAKE.CORTEX_USER database role on your Snowflake account. Each trainee uses a personal trial account where you are the administrator — the facilitator has no access to grant roles for you.

If you have not already, run snowflake/sql/setup/05_cortex_access.sql as ACCOUNTADMIN (same step as Module 6). See Snowflake Cortex ML Functions.

2. Verify Cortex ML access

Per Cortex ML prerequisites, confirm your role has the CORTEX_USER database role:

SHOW GRANTS TO ROLE DE_WORKSHOP_ROLE;
-- Expect: USAGE on database role SNOWFLAKE.CORTEX_USER

Functional check — run Step A0 from ml/snowflake/01_cortex_ml.sql to create the forecast input view, then verify:

SELECT COUNT(*), MIN(pickup_hour_ts), MAX(pickup_hour_ts)
FROM DE_MASTERCLASS.DE_01_ALICE_SQL_GOLD.V_HOURLY_TRIPS_BY_BOROUGH;

Expected: multiple boroughs and hourly timestamps spanning the dataset.

NoteNot the same as Module 6 LLM

Module 9 uses Cortex ML Functions (ML.FORECAST, ML.ANOMALY_DETECTION). Module 6 uses Cortex LLM functions such as AI_COMPLETE. You do not need AI_COMPLETE for this module.

3. Verify ML training data

-- Verify Silver enriched table exists with data
SELECT
    COUNT(*)                          AS total_rows,
    COUNT(CASE WHEN PAYMENT_TYPE_DESC = 'Credit card' THEN 1 END) AS credit_card_rows,
    AVG(TIP_AMOUNT)                   AS avg_tip
FROM DE_MASTERCLASS.DE_01_ALICE_SQL_SILVER.SILVER_NYC_TAXI_ENRICHED;

Expected: total_rows > 200,000 and credit_card_rows > 100,000.

4. Snowpark ML — local Python (optional)

If you run ml/snowflake/02_snowpark_ml_tip_prediction.py from Codespaces, use .env from Exercise: dbt § Configure .env — do not hardcode credentials in the script.

TipLocal / Codespaces

Install ML deps with uv pip install --system ".[ml]" (or use a Codespace — [ml] is pre-installed in the devcontainer).

Before running the script:

bash .devcontainer/setup-environment.sh   # loads .env — all variables should show ✅
python ml/snowflake/02_snowpark_ml_tip_prediction.py

If any variable is ❌, complete Exercise: dbt § Configure .env first.

Manual connection (local fallback only — prefer .env):

from snowflake.snowpark import Session

connection_params = {
    "account":   "your-account-id",
    "user":      "your-username",
    "password":  "your-password",
    "warehouse": "DE_WORKSHOP_WH",
    "database":  "DE_MASTERCLASS",
    "schema":    f"{ATTENDEE_ID}_SILVER",
    "role":      "SYSADMIN"
}
session = Session.builder.configs(connection_params).create()
NoteIn Snowsight notebooks

Snowpark Python is pre-installed in Snowsight notebooks — no local setup needed. Open your Snowflake account → Projects → Notebooks → create a new notebook.

dbt Setup for ml_features Model

The dbt ML model (dbt_project/models/ml/ml_features_tip_prediction.sql) materialises the feature table in {ATTENDEE_ID}_DBT_GOLD (+schema: dbt_gold — isolated from batch _SQL_GOLD KPIs).

Requires Exercise: dbt § Configure .env from Module 4.

Run it with your existing dbt Snowflake profile:

cd dbt_project/
dbt run --target snowflake --select ml_features_tip_prediction
dbt test --target snowflake --select ml_features_tip_prediction

Verify:

SELECT COUNT(*), AVG(fare_amount), AVG(target_tip_amount)
FROM DE_MASTERCLASS.DE_01_ALICE_DBT_GOLD.ML_FEATURES_TIP_PREDICTION;

Day-of verify list

Before Module 9 starts:

  • Databricks cluster running ML runtime (check version ends in ML)
  • import sklearn, mlflow succeeds in a notebook cell
  • Snowflake: SILVER_NYC_TAXI_ENRICHED has > 100k credit card rows
  • Snowflake: V_HOURLY_TRIPS_BY_BOROUGH view exists (Step A0 in 01_cortex_ml.sql)
  • Cortex ML access verified (SHOW GRANTS includes SNOWFLAKE.CORTEX_USER on your workshop role)
  • 05_cortex_access.sql run if Module 6 was skipped
  • (dbt or Snowpark ML in Codespace) Exercise: dbt § Configure .env done — bash .devcontainer/setup-environment.sh shows all ✅
Troubleshooting
Issue Solution
ModuleNotFoundError: sklearn on Databricks Switch to ML Runtime (cluster edit → Runtime version)
MLflow experiment not visible Click Experiments in sidebar; first run creates it automatically
Cortex ACCESS_DENIED Re-run snowflake/sql/setup/05_cortex_access.sql as ACCOUNTADMIN; use a role that received SNOWFLAKE.CORTEX_USER
Cortex ML.FORECAST returns 0 rows Run Step A0 to create V_HOURLY_TRIPS_BY_BOROUGH; verify Silver enriched has borough data
Snowpark Session.builder fails Check account ID format: el30551.west-europe.azure (from Snowsight URL or account details)
dbt model fails with missing source Run Silver and Gold batch pipeline first (Modules 2–4)
tip_amount all zeros in predictions Filter to payment_type_desc = 'Credit card' (matches Silver pipeline casing) — cash tips are always 0
dbt accepted_values fails on payment type Use 'Credit card' exactly — see _ml_features.yml
Row counts differ between dbt features and notebook training Confirm notebook uses the same filters as dbt (fare_amount < 200, tip_amount < 100, etc.)