Glossary

Key terms and concepts

title: “Glossary” subtitle: “Key terms and concepts” —

A

ADLS2 (Azure Data Lake Storage Gen2)
Azure’s scalable cloud storage service optimized for big data analytics. Supports hierarchical namespace and abfss:// protocol. Workshop source: mhpdeworkshopsa/nyc-taxi-data.
ADLS2 SAS token
Shared Access Signature for the workshop ADLS2 container (nyc-taxi-data). Used for Snowflake external stage access in trial accounts.
ai_query() (Databricks)
Databricks SQL function for LLM inference over table data (Module 6). Complements notebook Assistant and Genie — different from Snowflake AI_COMPLETE().
Aiven Kafka
Managed Apache Kafka broker used in Module 8 (optional). Facilitator runs a shared free-tier cluster; attendees consume the user-activity topic (Avro + Karapace Schema Registry). Separate dataset from NYC Taxi batch bronze — story uses taxi GPS; lab uses user-activity for a live topic per attendee.
Asset Bundles (DABs)
Databricks’ infrastructure-as-code tool for deploying pipelines, jobs, and notebooks via CLI. Pairs with LSDP and Lakeflow Jobs in production patterns. See databricks bundle deploy.

B

Bronze Layer
First layer of medallion architecture. Raw data ingested as-is from source, with metadata added. No business transformations. Metadata column names vary by ingest path — see Bronze schema.

C

Catalog (Unity Catalog)
Top-level namespace in Databricks: Catalog > Schema > Table. Provides centralized governance, auditing, and lineage.
Checkpoint
A file on cloud storage that records the exact offset a streaming query last processed. Enables exactly-once semantics and fault recovery in Structured Streaming.
COPY INTO
Snowflake SQL command for bulk loading data from external stages into tables. Efficient for Parquet, CSV, JSON.
Cortex AI (LLM)
Snowflake LLM and assistant features for Module 6: AI_COMPLETE() (replaces legacy SNOWFLAKE.CORTEX.COMPLETE()), AI_EXTRACT(), Snowflake CoCo (formerly Cortex Code / Copilot in Workspaces), Cortex Analyst, Cortex Search. Not the same as Cortex ML (Module 9).
Snowflake CoCo
Snowflake-native AI coding agent (rebrand of Cortex Code, 2026). In Snowsight Workspaces, use Cmd+I / Ctrl+I for NL → SQL/Python assistance with warehouse context and RBAC. Same workshop role as the old “Cortex Code in Snowsight” exercise — see Module 6 Exercise 4. CLI/Desktop are optional; not required for the lab.
Cortex Analyst
Snowflake service that translates natural language questions into SQL against governed semantic models (YAML definitions of tables and metrics).
Cortex ML
Snowflake managed ML SQL functions for Module 9: ML.FORECAST, ML.ANOMALY_DETECTION, and related time-series / classification APIs. Requires SNOWFLAKE.CORTEX_USER on your role. No custom model artifacts — contrast with Databricks sklearn + MLflow.
Cortex Search
Snowflake hybrid search service (semantic + keyword) over unstructured and structured content. Enables RAG patterns within Snowflake.
Cross-validation
ML evaluation technique that splits data into k folds, training on k-1 and testing on 1, rotating through all folds. More robust than a single train/test split.

D

Data Leakage
Including information in ML features that wouldn’t be available at prediction time or mathematically encodes the target. The #1 ML mistake (e.g., total_amount includes tip_amount).
dbt (data build tool)
Transformation framework that turns SQL SELECT statements into managed tables/views with testing, documentation, and lineage. Does not ingest — reads existing Bronze via source().
Delta Lake
Open-source storage format providing ACID transactions, time travel, and schema evolution on top of Parquet files.
dynamic_table (dbt)
dbt materialization type on the Snowflake adapter only (materialized: dynamic_table). Module 8 stretch — declarative refresh similar to native Dynamic Tables; lands in {attendee}_DBT_STREAMING, isolated from Snowflake-native _STREAMING objects.
Dynamic Table (Snowflake)
Snowflake object defined by a SQL SELECT that refreshes automatically based on TARGET_LAG. Declarative streaming — you write SQL, Snowflake manages incremental refresh. Can also be created via dbt dynamic_table materialization.

E

ELT (Extract, Load, Transform)
Pattern where raw data is loaded first, then transformed in-place within the data warehouse. Used by all three pipelines in this training.
Exactly-once Semantics
Guarantee that each event is processed exactly once, even during failures. Achieved in Structured Streaming via checkpoint files + Delta Lake transaction log.
External Stage
Snowflake object pointing to a cloud storage location (ADLS2, S3, GCS) for reading external files.

F

Feature Engineering
Process of creating input variables (features) for ML models from raw data. In this workshop each platform has its own path: Databricks sklearn on silver_nyc_taxi_enriched, Snowflake Cortex ML on Gold/time-series views, optional Snowpark ML, and optional dbt ml_features_tip_prediction in {attendee}_DBT_GOLD — dbt is a contract/testing layer, not the single source all trainers consume.

G

Genie Space
Databricks AI/BI feature. Natural language interface over data tables — ask questions in plain English, get SQL + results (Module 6).
Gold Layer
Third layer of medallion architecture. Pre-aggregated business KPIs and metrics optimized for reporting and dashboards. This training materialises 12 KPI tables.

I

Incremental Materialization
dbt strategy that appends or merges only new rows instead of rebuilding the entire table. Requires is_incremental() logic and a unique_key. Advanced pattern — base workshop models use table / view materializations only.

J

Jinja
Templating engine embedded in dbt SQL files. Enables conditional compilation (target.type), reusable macros, ref()/source() dependency management, and configuration blocks.

K

KPI (Key Performance Indicator)
Pre-computed business metric. This training computes 12 KPIs from taxi trip data (trips by hour, top zones, revenue, etc.).

L

Lakeflow Jobs
Databricks job scheduler (formerly Workflows). Runs notebooks, LSDP pipelines, Python scripts, dbt tasks, and SQL as multi-task DAGs with dependencies, retries, and notifications.
Lakeflow Spark Declarative Pipelines (LSDP)
Databricks declarative ETL framework for production (formerly Delta Live Tables / DLT, renamed 2025). Workshop code uses from pyspark import pipelines as dp with @dp.materialized_view, @dp.expect_or_drop, and spark.read.table(). Legacy import dlt / @dlt.table still works on many workspaces.

M

Macro (dbt)
Reusable SQL function defined in .sql files under macros/. Similar to functions in Python — accepts parameters and returns SQL snippets. Example: { time_of_day('pickup_hour') }.
Materialization
Strategy for how a dbt model becomes a database object. Core types: view, table, incremental, ephemeral, materialized_view (Databricks adapter). Snowflake adapter adds dynamic_table for Module 8 streaming.
Medallion Architecture
Multi-layer data design pattern: Bronze (raw) → Silver (cleaned) → Gold (aggregated). Also called “multi-hop architecture.”
Metric Views (Unity Catalog)
Databricks UC object (public preview, 2025) for defining reusable business metrics — measure, dimensions, and filters — once in catalog metadata. Consumers query the same definition from dashboards, Genie, and alerts instead of re-implementing KPI SQL. Workshop uses physical Gold kpi_* tables, not Metric Views; see Module 2 callout and Gold Layer.
MLflow
Open-source platform for ML lifecycle management. Tracks experiments (hyperparameters, metrics, model artifacts). Databricks includes MLflow autolog for automatic experiment capture.
Model (dbt)
A SQL SELECT statement that dbt materializes as a table or view. Each .sql file in models/ is one model.

P

Pipeline expectations (@dp.expect_or_drop)
LSDP decorators that declare data quality rules on pipeline tables. Violations can be dropped or flagged with metrics in the pipeline event log — declarative alternative to manual WHERE filters (Module 5).
Power BI
Microsoft BI tool for the five-page KPI dashboard (Module 7 lab). Connects in Import mode to Gold tables from any completed batch path (Databricks, Snowflake, or dbt).
PySpark
Python API for Apache Spark. Used in Databricks notebooks for distributed data processing.

R

ref() (dbt)
dbt function that creates a dependency between models. { ref('my_model') } references another model and builds the DAG.
Relay consumer
Facilitator-managed Python script (streaming/snowflake/00_relay_consumer.py) that reads Aiven Kafka and lands NDJSON files on ADLS2 for Snowflake Snowpipe. Used because Aiven free tier has no Kafka Connect (Module 8).

S

Schema tracks (workshop)
Per-attendee isolated schemas so pipelines do not overwrite each other: Databricks {attendee_id}_bronze|silver|gold; Snowflake _SQL_*, _SP_*, _DBT, _DBT_GOLD, _STREAMING, _DBT_STREAMING. See recommended workshop tracks.
Silver Layer
Second layer of medallion architecture. Cleaned, validated, enriched data with derived metrics. Trustworthy data for analysis. Outputs: silver_nyc_taxi_cleaned + silver_nyc_taxi_enriched.
Slim CI
dbt testing pattern that only tests changed models + their downstream dependencies instead of the entire project. Uses --select state:modified+.
Snowpark
Snowflake’s Python DataFrame API. Runs Python code natively on Snowflake compute. API is similar to PySpark. Used for batch ingest/transform and optional Snowpark ML (Module 9).
Snowflake Intelligence
Umbrella brand (announced 2025) bundling Cortex LLM, Analyst, Search, and ML functions into a unified AI layer.
Snowpipe
Snowflake continuous loader that ingests new files from a stage into a table. Module 8 uses it on the relay-consumer NDJSON path under streaming/user-activity/.
source() (dbt)
dbt function referencing a raw table not managed by dbt. Example: { source('bronze', 'nyc_taxi_trips') }. dbt does not read ADLS2 directly — your notebook or Snowpark job must create Bronze first.
Stream (Snowflake)
Snowflake object that tracks changes (inserts, updates, deletes) on a table. Used for change data capture (CDC).
Structured Streaming
Databricks / Spark API for processing unbounded data in micro-batches. Module 8 reads Aiven Kafka with readStream and writes append-only Delta bronze tables.

T

Task (Snowflake)
Snowflake object that runs SQL on a schedule or when triggered by a condition. Used with Streams for event-driven pipelines.
Three Constraints
YellowLine NYC decision framework (Module 7): Cost, Performance, and Compliance — used to recommend a platform mix for Marcus, not a single “winner.” See Module 7 — Comparison & Wrap-up.

U

Unity Catalog
Databricks’ centralized governance layer. Manages catalogs, schemas, tables, permissions, lineage, and data sharing.

V

Virtual Warehouse (Snowflake)
Snowflake compute resource. Auto-suspends when idle, auto-resumes on query. Sizes from X-Small to 6X-Large.

W

Watermark
In streaming, a threshold that tells the engine to ignore events older than X minutes. Enables window finalization and state memory cleanup. Tradeoff: shorter = lower latency but more dropped late events.

Y

YellowLine NYC
Fictional NYC yellow-taxi operator used as the training client story (inspired by TLC public trip data, not a real company). Marcus Chen (Operations Manager) works for YellowLine NYC; MHP is the consulting vendor engaged to build analytics. On-screen dispatch UI and Marcus’s badge use YellowLine branding — not MHP.