1 — ADLS2 Storage

Pre-class Checklist — 2 weeks before

TRAINER ONLY

2 weeks before Step 1 of 9

2 Weeks Before — Infrastructure

Note

Azure resource group — ADLS2 and Databricks are pre-provisioned together under 1000_data_engineering_workshop (subscription MHP Resort Consulting Services, tenant mhpdev.onmicrosoft.com).

Set this up first — Databricks 00_setup.py and Snowflake external stages depend on the ADLS2 storage account access key and ADLS2 SAS token from this page.

ImportantTrip data month (not hardcoded)

Set the delivery month once in workshop-data.json and Workshop data config before download/upload.

Filename pattern: yellow_tripdata_{YYYY}-{MM}.parquet (e.g. yellow_tripdata_2024-10.parquet when year=2024, month=10).

Pipelines read the raw/trips/ folder — upload only the Parquet file for the configured month unless you intend to load multiple months.

Azure / ADLS2 Storage (mhpdeworkshopsa)

Pre-provisioned — account already exists; do not create a new one.

Item	Value
Storage account	mhpdeworkshopsa
Resource group	1000_data_engineering_workshop
Location	Germany West Central
Subscription	MHP Resort Consulting Services (ba826c91-8e52-4e07-ac7c-538858bbc813)
Azure tenant	mhpdev.onmicrosoft.com

• Storage account accessible
  • Where: Azure Portal → resource group 1000_data_engineering_workshop → click mhpdeworkshopsa
  • Verify: Overview blade loads; Status = Available
  • Doc: Azure storage account overview
• Container nyc-taxi-data exists (create only if missing)
  • Where: Storage account mhpdeworkshopsa → left menu Data storage → Containers
  • Configure: If missing: + Container → Name nyc-taxi-data → Create
  • Doc: Create a blob container
• Workshop month set in workshop-data.json (tripDataYear, tripDataMonth)
  • Where: Repo file workshop-2026-v2/workshop-data.json (local editor or GitHub web UI)
  • Configure: Set tripDataYear and tripDataMonth to match the TLC Parquet file you will upload (e.g. "2024", "10")
  • Doc: Workshop data config
• TLC data downloaded from NYC TLC Trip Record Data
  • Where: TLC website → Yellow Taxi Trip Records → select year → download Parquet (not CSV)
  • Doc: NYC TLC trip record data
  • yellow_tripdata_{YYYY}-{MM}.parquet (month from workshop-data.json)
  • taxi_zone_lookup.csv (Auxiliary / zone lookup — same TLC page or workshop repo if cached)
• Uploaded to ADLS2 mhpdeworkshopsa / nyc-taxi-data
  • Where: Storage account mhpdeworkshopsa → Storage browser (or Data storage → Containers → nyc-taxi-data) → navigate/create folders → Upload
  • Configure: Upload paths below (create raw/trips/ and raw/lookup/ folders if missing)
  • Doc: Upload blobs with Storage browser
  • raw/trips/yellow_tripdata_{YYYY}-{MM}.parquet (same month as workshop-data.json)
  • raw/lookup/taxi_zone_lookup.csv (exact filename)
• Container paths verified in Portal
  • Where: Storage browser → Blob containers → nyc-taxi-data
  • Verify: raw/trips/ contains the workshop-month Parquet; raw/lookup/taxi_zone_lookup.csv exists
  • Doc: Browse blobs
• ADLS2 storage account access key (Databricks)
  • Where: Storage account mhpdeworkshopsa → Security + networking → Access keys
  • Configure: Click Show on key1 → Copy (store in password manager — distribute verbally on Module 2)
  • Verify: Paste into trainer dry-run of 00_setup.py → STORAGE_ACCOUNT_KEY cell → bronze read succeeds
  • Doc: Manage storage account access keys
• ADLS2 SAS token (Snowflake external stage)
  • Where: Storage browser → nyc-taxi-data → top toolbar Generate SAS (or container ⋯ → Generate SAS)
  • Configure: Signing method Account key · Permissions Read + List only · Start/ expiry so token ends workshop date + 2 days · Generate SAS token and URL → copy the SAS token query string (?sv=...) immediately
  • Verify: Token saved in cohort settings / credentials sheet; not committed to Git
  • Doc: Create a service SAS (portal)
• Verify: Databricks spark.read.parquet(...) works with new key
  • Where: Databricks workspace → trainer cluster → run 00_setup.py / 01_bronze_ingestion.py
  • Verify: Bronze table row count > 0
  • Doc: Read files from Azure Data Lake Storage
• Unity Catalog external location — required for the serverless Lakeflow pipeline
  • Why: Serverless compute blocks fs.azure.account.key injection; the pipeline must read ADLS2 through a UC-managed credential instead.
  • Steps (one-time, metastore-admin required):

    1. Create an Access Connector (Azure Portal → create resource → Access Connector for Azure Databricks in the same resource group → assign Storage Blob Data Contributor on mhpdeworkshopsa)

    2. Create a storage credential in Databricks (Catalog Explorer → External Data → Storage Credentials → Add → Managed Identity → paste the Access Connector resource ID)

    3. Create an external location (Catalog Explorer → External Data → External Locations → Add):

       • URL: abfss://nyc-taxi-data@mhpdeworkshopsa.dfs.core.windows.net/
       • Credential: the one created above

    4. Grant attendees READ FILES:

       GRANT READ FILES ON EXTERNAL LOCATION `<your-external-location-name>`
         TO `account users`;

  • Verify: In a serverless notebook: dbutils.fs.ls("abfss://nyc-taxi-data@mhpdeworkshopsa.dfs.core.windows.net/raw/trips/") returns the Parquet file
  • Doc: Create an external location (Unity Catalog)
• Verify: Snowflake LIST @...nyc_taxi_trips_stage returns files with ADLS2 SAS token
  • Where: Snowsight → Projects → Workspaces → run LIST @TR_01_JUEWEI_SQL_BRONZE.nyc_taxi_trips_stage (your trainer ID + _SQL_BRONZE schema)
  • Verify: Parquet file from raw/trips/ appears in output
  • Doc: Snowflake external stages (Azure)

---

→ Next 2 — Databricks workspace Needs ADLS2 storage account access key