CI/CD for Databricks with Azure DevOps: Service Principals, DABs Deployment, YAML Pipelines, Notebook Testing, Unity Catalog Promotion, Workspace-First Development, and Production Deployment Patterns

Leave a Comment / Azure, Data Engineering, DevOps

Table of Contents

Our DABs post covered Databricks Asset Bundles — the YAML-based project format that defines notebooks, jobs, pipelines, and clusters as code. Our Git Integration post covered Databricks Repos and GitHub Actions. This post connects everything: deploying DABs through Azure DevOps YAML pipelines with service principals, testing, Unity Catalog promotion, and approval gates — the production-grade CI/CD that enterprise data teams use.

Analogy — A car factory assembly line. Developers design the car in the engineering lab (dev workspace). When the design is finalized (merged to main), it enters the factory assembly line (Azure DevOps pipeline). Station 1 (CI) inspects the blueprints for errors (validate, lint, test). Station 2 (deploy to dev) builds a prototype. Station 3 (deploy to staging) builds a test model that QA drives. Station 4 (deploy to prod) builds the production car — but only after the factory manager signs off (approval gate). The assembly line is automated end to end. No human touches the production car directly.

CI/CD for Databricks — The Big Picture

The Databricks CI/CD landscape in 2026:

  What gets deployed:
    - Notebooks (Python, SQL, Scala)
    - Jobs and Workflows (scheduling, dependencies)
    - Lakeflow Declarative Pipelines (medallion layers, quality expectations)
    - Cluster configurations (shared clusters, job clusters)
    - Unity Catalog objects (catalogs, schemas, grants)
    - Python wheel packages (shared libraries)

  How it gets deployed:
    Databricks Asset Bundles (DABs) -- the recommended approach
      databricks bundle validate  → check YAML syntax
      databricks bundle deploy    → deploy to target workspace
      databricks bundle run       → trigger a job run

  Where the pipeline runs:
    Azure DevOps YAML Pipelines (this post)
    GitHub Actions (covered in our Git Integration post)

  Authentication:
    Service Principal + Azure AD → Databricks workspace
    Service Connection → Azure DevOps to Azure

Setting Up Service Principals for Databricks CI/CD

A service principal (SP) is a non-human identity that Azure DevOps pipelines use to authenticate with Databricks. You need one SP per environment.

Step-by-step setup:

  1. CREATE APP REGISTRATION IN AZURE AD
     Azure Portal → Azure Active Directory → App registrations → New
     Name: "sp-databricks-dev" (or sp-databricks-prod)
     Create a client secret: Certificates & Secrets → New Client Secret
     Note: Application (client) ID, Directory (tenant) ID, Client Secret

  2. ADD SP TO DATABRICKS WORKSPACE
     Databricks workspace → Admin Console → Service Principals → Add
     Select the App Registration you created
     Grant workspace access (User or Admin)

  3. GRANT UNITY CATALOG PERMISSIONS
     GRANT USE CATALOG ON CATALOG dev_catalog TO `sp-databricks-dev`;
     GRANT USE SCHEMA ON SCHEMA dev_catalog.bronze TO `sp-databricks-dev`;
     GRANT ALL PRIVILEGES ON SCHEMA dev_catalog.bronze TO `sp-databricks-dev`;

  4. STORE CREDENTIALS IN AZURE DEVOPS
     Pipelines → Library → Variable Group: "databricks-dev-credentials"
     - DATABRICKS_HOST = https://adb-123.azuredatabricks.net
     - ARM_CLIENT_ID = <from step 1>
     - ARM_CLIENT_SECRET = <from step 1> (mark as secret)
     - ARM_TENANT_ID = <from step 1>

  5. Repeat for staging and prod with separate SPs
# Authenticate Databricks CLI using service principal in pipeline
steps:
  - script: |
      export DATABRICKS_HOST=$(DATABRICKS_HOST)
      export ARM_CLIENT_ID=$(ARM_CLIENT_ID)
      export ARM_CLIENT_SECRET=$(ARM_CLIENT_SECRET)
      export ARM_TENANT_ID=$(ARM_TENANT_ID)
      databricks auth login --host $(DATABRICKS_HOST)
      databricks current-user me
    displayName: 'Authenticate with Databricks'
    env:
      ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)

Installing the Databricks CLI in Pipelines

# Install Databricks CLI v2 (required for DABs)
steps:
  - script: |
      curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
      databricks --version
    displayName: 'Install Databricks CLI'

  # Alternative: install via pip
  - script: |
      pip install databricks-cli
      databricks --version
    displayName: 'Install Databricks CLI (pip)'

Databricks Repos Integration with Azure DevOps

Databricks Repos syncs Git repositories directly into the Databricks workspace. This is the development model: engineers edit notebooks in the workspace, commit from Repos, and CI/CD deploys via DABs.

Two development workflows:

  Workflow 1: Workspace-First (most common)
    1. Developer opens Databricks workspace
    2. Creates/edits notebooks in Repos (connected to Azure DevOps Git)
    3. Commits and pushes from the Repos UI
    4. Creates a PR in Azure DevOps
    5. CI pipeline validates (lint, test)
    6. Merge to main → CD pipeline deploys via DABs

  Workflow 2: IDE-First (local development)
    1. Developer clones repo locally (VS Code, PyCharm)
    2. Edits notebooks as .py files
    3. Tests locally with databricks-connect or pytest
    4. Pushes to Azure DevOps
    5. Creates PR → CI validates → merge → CD deploys

  Most data engineering teams use Workflow 1 for notebook development
  and Workflow 2 for shared Python libraries and Terraform.
# Update Databricks Repos in a pipeline (sync workspace with latest code)
steps:
  - script: |
      databricks repos update \
        --path /Repos/production/data-platform \
        --branch main
    displayName: 'Sync Repos to main branch'
    env:
      DATABRICKS_HOST: $(DATABRICKS_HOST)
      ARM_CLIENT_ID: $(ARM_CLIENT_ID)
      ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
      ARM_TENANT_ID: $(ARM_TENANT_ID)

Deploying DABs Through Azure DevOps Pipelines

The core deployment command is simple: databricks bundle deploy --target <environment>. The pipeline wraps this with authentication, validation, and approval gates.

# Basic DABs deployment step
steps:
  - script: |
      cd databricks-project/
      databricks bundle validate --target dev
      databricks bundle deploy --target dev --auto-approve
    displayName: 'Deploy DABs to Dev'
    env:
      DATABRICKS_HOST: $(DATABRICKS_HOST)
      ARM_CLIENT_ID: $(ARM_CLIENT_ID)
      ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
      ARM_TENANT_ID: $(ARM_TENANT_ID)

The databricks.yml Configuration for Multi-Environment

# databricks.yml -- single file, multiple targets
bundle:
  name: data-platform-pipelines

variables:
  catalog_name:
    description: "Unity Catalog name"
  environment:
    description: "Environment identifier"

# Shared resources (same across all environments)
resources:
  jobs:
    daily_ingestion:
      name: "${var.environment}_daily_ingestion"
      tasks:
        - task_key: extract
          notebook_task:
            notebook_path: ./notebooks/01_extract.py
          new_cluster:
            spark_version: "15.4.x-scala2.12"
            node_type_id: "Standard_DS3_v2"
            num_workers: 2

        - task_key: transform
          depends_on:
            - task_key: extract
          notebook_task:
            notebook_path: ./notebooks/02_transform.py
            base_parameters:
              catalog: "${var.catalog_name}"

        - task_key: load
          depends_on:
            - task_key: transform
          notebook_task:
            notebook_path: ./notebooks/03_load.py
            base_parameters:
              catalog: "${var.catalog_name}"

      schedule:
        quartz_cron_expression: "0 0 6 * * ?"
        timezone_id: "America/Toronto"

  pipelines:
    medallion_pipeline:
      name: "${var.environment}_medallion"
      target: "${var.catalog_name}.bronze"
      catalog: "${var.catalog_name}"
      libraries:
        - notebook:
            path: ./pipelines/bronze_layer.py
        - notebook:
            path: ./pipelines/silver_layer.py
        - notebook:
            path: ./pipelines/gold_layer.py

# Environment-specific targets
targets:
  dev:
    mode: development
    default: true
    workspace:
      host: https://adb-dev-123.azuredatabricks.net
    variables:
      catalog_name: dev_catalog
      environment: dev
    resources:
      jobs:
        daily_ingestion:
          schedule: null     # No schedule in dev (manual runs only)
          tasks:
            - task_key: extract
              new_cluster:
                num_workers: 1    # Smaller cluster in dev

  staging:
    workspace:
      host: https://adb-staging-456.azuredatabricks.net
    variables:
      catalog_name: staging_catalog
      environment: staging

  prod:
    mode: production
    workspace:
      host: https://adb-prod-789.azuredatabricks.net
    variables:
      catalog_name: prod_catalog
      environment: prod
    resources:
      jobs:
        daily_ingestion:
          tasks:
            - task_key: extract
              new_cluster:
                num_workers: 4    # Larger cluster in prod

CI Pipeline — Validate and Test on Every PR

# pipelines/databricks-ci.yml
trigger: none
pr:
  branches:
    include:
      - main
  paths:
    include:
      - databricks-project/**

pool:
  vmImage: 'ubuntu-latest'

variables:
  - group: 'databricks-dev-credentials'

steps:
  # Install tools
  - script: |
      curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
      pip install pytest ruff pyspark
    displayName: 'Install tools'

  # Validate bundle YAML
  - script: |
      cd databricks-project/
      databricks bundle validate --target dev
    displayName: 'Validate DABs configuration'
    env:
      DATABRICKS_HOST: $(DATABRICKS_HOST)
      ARM_CLIENT_ID: $(ARM_CLIENT_ID)
      ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
      ARM_TENANT_ID: $(ARM_TENANT_ID)

  # Lint Python code
  - script: |
      ruff check databricks-project/notebooks/ --output-format=github
      ruff check databricks-project/pipelines/ --output-format=github
    displayName: 'Lint notebooks'

  # Run unit tests
  - script: |
      cd databricks-project/
      pytest tests/unit/ -v --junitxml=test-results.xml
    displayName: 'Run unit tests'

  # Publish test results
  - task: PublishTestResults@2
    inputs:
      testResultsFiles: 'databricks-project/test-results.xml'
    displayName: 'Publish test results'
    condition: always()

CD Pipeline — Deploy Through Environments

# pipelines/databricks-cd.yml
trigger:
  branches:
    include:
      - main
  paths:
    include:
      - databricks-project/**

stages:
  # Stage 1: Deploy to Dev
  - stage: DeployDev
    displayName: 'Deploy to Dev'
    pool:
      vmImage: 'ubuntu-latest'
    variables:
      - group: 'databricks-dev-credentials'
    jobs:
      - deployment: DeployDABsDev
        environment: 'dev'
        strategy:
          runOnce:
            deploy:
              steps:
                - checkout: self
                - script: |
                    curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
                  displayName: 'Install CLI'
                - script: |
                    cd databricks-project/
                    databricks bundle deploy --target dev --auto-approve
                  displayName: 'Deploy to Dev'
                  env:
                    DATABRICKS_HOST: $(DATABRICKS_HOST)
                    ARM_CLIENT_ID: $(ARM_CLIENT_ID)
                    ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
                    ARM_TENANT_ID: $(ARM_TENANT_ID)

  # Stage 2: Deploy to Staging (requires approval)
  - stage: DeployStaging
    displayName: 'Deploy to Staging'
    dependsOn: DeployDev
    pool:
      vmImage: 'ubuntu-latest'
    variables:
      - group: 'databricks-staging-credentials'
    jobs:
      - deployment: DeployDABsStaging
        environment: 'staging'
        strategy:
          runOnce:
            deploy:
              steps:
                - checkout: self
                - script: |
                    curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
                  displayName: 'Install CLI'
                - script: |
                    cd databricks-project/
                    databricks bundle deploy --target staging --auto-approve
                  displayName: 'Deploy to Staging'
                  env:
                    DATABRICKS_HOST: $(DATABRICKS_HOST)
                    ARM_CLIENT_ID: $(ARM_CLIENT_ID)
                    ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
                    ARM_TENANT_ID: $(ARM_TENANT_ID)

  # Stage 3: Integration test in Staging
  - stage: IntegrationTest
    displayName: 'Integration Tests'
    dependsOn: DeployStaging
    pool:
      vmImage: 'ubuntu-latest'
    variables:
      - group: 'databricks-staging-credentials'
    jobs:
      - job: RunTests
        steps:
          - script: |
              curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
            displayName: 'Install CLI'
          - script: |
              cd databricks-project/
              databricks bundle run daily_ingestion --target staging
            displayName: 'Run job in Staging'
            env:
              DATABRICKS_HOST: $(DATABRICKS_HOST)
              ARM_CLIENT_ID: $(ARM_CLIENT_ID)
              ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
              ARM_TENANT_ID: $(ARM_TENANT_ID)

  # Stage 4: Deploy to Production (requires 2 approvals)
  - stage: DeployProd
    displayName: 'Deploy to Production'
    dependsOn: IntegrationTest
    condition: |
      and(
        succeeded(),
        eq(variables['Build.SourceBranch'], 'refs/heads/main')
      )
    pool:
      vmImage: 'ubuntu-latest'
    variables:
      - group: 'databricks-prod-credentials'
    jobs:
      - deployment: DeployDABsProd
        environment: 'production'
        strategy:
          runOnce:
            deploy:
              steps:
                - checkout: self
                - script: |
                    curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh
                  displayName: 'Install CLI'
                - script: |
                    cd databricks-project/
                    databricks bundle deploy --target prod --auto-approve
                  displayName: 'Deploy to Production'
                  env:
                    DATABRICKS_HOST: $(DATABRICKS_HOST)
                    ARM_CLIENT_ID: $(ARM_CLIENT_ID)
                    ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
                    ARM_TENANT_ID: $(ARM_TENANT_ID)

Testing Notebooks in CI/CD

Unit Tests (Run Without Databricks)

# tests/unit/test_transformations.py
# These tests run on the CI agent (no Databricks cluster needed)
import pytest
from pyspark.sql import SparkSession
from pyspark.sql.types import StructType, StructField, StringType, DecimalType

@pytest.fixture(scope="session")
def spark():
    return SparkSession.builder.master("local[*]").appName("tests").getOrCreate()

def test_clean_customer_name(spark):
    """Test that customer names are cleaned correctly."""
    from notebooks.utils.transformations import clean_customer_name

    data = [("  JOHN DOE  ",), ("jane smith",), (None,)]
    df = spark.createDataFrame(data, ["name"])
    result = clean_customer_name(df, "name")
    names = [r.name for r in result.collect()]

    assert names[0] == "John Doe"
    assert names[1] == "Jane Smith"
    assert names[2] is None

def test_filter_valid_orders(spark):
    """Test that invalid orders are filtered out."""
    from notebooks.utils.transformations import filter_valid_orders

    schema = StructType([
        StructField("order_id", StringType()),
        StructField("amount", DecimalType(10, 2)),
    ])
    data = [("1", 100.00), ("2", 0.00), ("3", -50.00), ("4", 250.00)]
    df = spark.createDataFrame(data, schema)
    result = filter_valid_orders(df)

    assert result.count() == 2  # Only orders with amount > 0

Integration Tests (Run on Databricks)

# tests/integration/test_pipeline.py
# These tests run ON Databricks (via databricks bundle run)

def test_bronze_table_populated():
    """Verify bronze table has data after ingestion."""
    count = spark.table("dev_catalog.bronze.raw_orders").count()
    assert count > 0, "Bronze table is empty after ingestion"

def test_silver_schema_valid():
    """Verify silver table has expected columns."""
    df = spark.table("dev_catalog.silver.orders_clean")
    expected_cols = {"order_id", "customer_name", "amount", "order_date", "processed_at"}
    actual_cols = set(df.columns)
    assert expected_cols.issubset(actual_cols), f"Missing columns: {expected_cols - actual_cols}"

def test_no_duplicates_in_silver():
    """Verify no duplicate order_ids in silver."""
    df = spark.table("dev_catalog.silver.orders_clean")
    total = df.count()
    distinct = df.select("order_id").distinct().count()
    assert total == distinct, f"Duplicates found: {total} rows, {distinct} distinct"

Unity Catalog Promotion Across Environments

Unity Catalog multi-environment strategy:

  Option 1: Separate catalogs (recommended)
    dev_catalog   → dev workspace
    staging_catalog → staging workspace
    prod_catalog  → prod workspace

    DABs target sets the catalog:
      targets.dev.variables.catalog_name = "dev_catalog"
      targets.prod.variables.catalog_name = "prod_catalog"

    Notebooks use the variable:
      catalog = dbutils.widgets.get("catalog")
      spark.sql(f"USE CATALOG {catalog}")

  Option 2: Same metastore, different catalogs
    All workspaces share one metastore
    Each workspace uses its own catalog
    Cross-environment reads possible (for testing)

  Option 3: Separate metastores per environment
    Complete isolation (regulated industries)
    No cross-environment access
    More operational overhead

  Most teams use Option 1: same metastore, separate catalogs.
# In databricks.yml, catalogs are parameterized:
resources:
  jobs:
    daily_ingestion:
      tasks:
        - task_key: transform
          notebook_task:
            base_parameters:
              catalog: "${var.catalog_name}"
              schema: "silver"

# Each target provides the correct catalog:
targets:
  dev:
    variables:
      catalog_name: dev_catalog
  prod:
    variables:
      catalog_name: prod_catalog

Workspace-First Development with Git-Backed Deployment

The recommended development workflow:

  1. DEVELOP in the workspace (fast iteration)
     Developer opens Databricks workspace
     Works on notebooks in Repos (linked to Azure DevOps repo)
     Tests interactively on a dev cluster
     Iterates quickly with real data

  2. COMMIT from the workspace
     Developer commits changes via Repos UI
     Pushes to a feature branch
     Creates a PR in Azure DevOps

  3. CI VALIDATES the PR
     Azure DevOps pipeline runs:
       databricks bundle validate
       ruff check (lint)
       pytest tests/unit/ (unit tests)
     PR shows green/red status

  4. MERGE triggers CD
     Approved PR merged to main
     CD pipeline deploys via DABs:
       Deploy to dev → Deploy to staging (with approval) → Deploy to prod

  5. PRODUCTION runs on prod workspace
     Jobs run on schedule (defined in databricks.yml)
     Developers NEVER touch prod workspace directly
     All changes go through Git → PR → CI → CD

  Key rule: workspace is for DEVELOPMENT, Git is for DEPLOYMENT
  No one deploys to prod by clicking buttons in the workspace

Complete End-to-End Pipeline

# pipelines/databricks-full.yml
# Complete CI/CD: validate → test → deploy dev → test staging → deploy prod
trigger:
  branches:
    include: [main]
  paths:
    include: [databricks-project/**]

pr:
  branches:
    include: [main]
  paths:
    include: [databricks-project/**]

variables:
  cliInstall: 'curl -fsSL https://raw.githubusercontent.com/databricks/setup-cli/main/install.sh | sh'

stages:
  # CI Stage: runs on every PR
  - stage: CI
    displayName: 'Validate and Test'
    condition: eq(variables['Build.Reason'], 'PullRequest')
    pool:
      vmImage: 'ubuntu-latest'
    variables:
      - group: 'databricks-dev-credentials'
    jobs:
      - job: Validate
        steps:
          - script: $(cliInstall)
            displayName: 'Install CLI'
          - script: |
              cd databricks-project/
              databricks bundle validate --target dev
            displayName: 'Validate bundle'
            env:
              DATABRICKS_HOST: $(DATABRICKS_HOST)
              ARM_CLIENT_ID: $(ARM_CLIENT_ID)
              ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
              ARM_TENANT_ID: $(ARM_TENANT_ID)
          - script: |
              pip install pytest ruff pyspark
              ruff check databricks-project/ --output-format=github
              pytest databricks-project/tests/unit/ -v --junitxml=results.xml
            displayName: 'Lint and Test'
          - task: PublishTestResults@2
            inputs:
              testResultsFiles: 'results.xml'
            condition: always()

  # CD Stages: run on merge to main
  - stage: DeployDev
    displayName: 'Deploy Dev'
    condition: and(succeeded(), ne(variables['Build.Reason'], 'PullRequest'))
    pool:
      vmImage: 'ubuntu-latest'
    variables:
      - group: 'databricks-dev-credentials'
    jobs:
      - deployment: Deploy
        environment: 'dev'
        strategy:
          runOnce:
            deploy:
              steps:
                - checkout: self
                - script: |
                    $(cliInstall)
                    cd databricks-project/
                    databricks bundle deploy --target dev --auto-approve
                  displayName: 'Deploy to Dev'
                  env:
                    DATABRICKS_HOST: $(DATABRICKS_HOST)
                    ARM_CLIENT_ID: $(ARM_CLIENT_ID)
                    ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
                    ARM_TENANT_ID: $(ARM_TENANT_ID)

  - stage: DeployProd
    displayName: 'Deploy Prod'
    dependsOn: DeployDev
    condition: and(succeeded(), eq(variables['Build.SourceBranch'], 'refs/heads/main'))
    pool:
      vmImage: 'ubuntu-latest'
    variables:
      - group: 'databricks-prod-credentials'
    jobs:
      - deployment: Deploy
        environment: 'production'
        strategy:
          runOnce:
            deploy:
              steps:
                - checkout: self
                - script: |
                    $(cliInstall)
                    cd databricks-project/
                    databricks bundle deploy --target prod --auto-approve
                  displayName: 'Deploy to Production'
                  env:
                    DATABRICKS_HOST: $(DATABRICKS_HOST)
                    ARM_CLIENT_ID: $(ARM_CLIENT_ID)
                    ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
                    ARM_TENANT_ID: $(ARM_TENANT_ID)

Common Mistakes

  1. Using personal access tokens (PATs) for CI/CD authentication. PATs are tied to a user account and expire. If the user leaves or the token expires, pipelines break. Always use service principals with Azure AD authentication for CI/CD. SPs have managed credentials, can be rotated without affecting people, and have scoped permissions.

  2. Not using –auto-approve in CI/CD pipelines. Without --auto-approve, the Databricks CLI prompts for confirmation on destructive actions. In a non-interactive pipeline, this causes the pipeline to hang indefinitely. Always include --auto-approve in your deploy commands.

  3. Deploying directly from the workspace to production. If developers create or modify jobs directly in the prod workspace UI, those changes are not tracked in Git, not reviewed, and not reproducible. Enforce the rule: workspace for development, Git for deployment. Only CI/CD pipelines deploy to staging and prod.

  4. Using the same service principal for all environments. A single SP with access to dev and prod means a misconfigured pipeline can modify production. Create dedicated SPs per environment with permissions scoped to that environment’s workspace and catalog.

  5. Not running databricks bundle validate before deploy. Validate catches YAML syntax errors, missing references, and configuration issues without making any changes to the workspace. Always run validate before deploy in your pipeline. It takes seconds and catches errors early.

  6. Hardcoding workspace URLs in pipeline YAML. Writing DATABRICKS_HOST: https://adb-prod-789.azuredatabricks.net in the YAML file means changing workspaces requires code changes. Store the host URL in a variable group per environment. The pipeline references $(DATABRICKS_HOST) regardless of the target.

  7. Not testing notebooks before deploying. Deploying untested notebooks to production is like shipping unreviewed code. Run unit tests (pytest with local PySpark) in CI and integration tests (databricks bundle run) in staging before promoting to production.

  8. Forgetting to set schedule to null in dev targets. If your databricks.yml defines a schedule for a job and you deploy to dev, the job starts running on schedule in dev — consuming compute credits unnecessarily. Override schedule: null in the dev target to prevent this.

Interview Questions

Q: How do you authenticate Azure DevOps pipelines with Databricks? A: Create a service principal (App Registration) in Azure AD with a client secret. Add the SP to the Databricks workspace as a service principal. Store the credentials (DATABRICKS_HOST, ARM_CLIENT_ID, ARM_CLIENT_SECRET, ARM_TENANT_ID) in an Azure DevOps variable group, ideally linked to Key Vault. In the pipeline, set these as environment variables. The Databricks CLI uses Azure AD authentication to connect to the workspace without personal access tokens.

Q: What is the recommended CI/CD workflow for Databricks? A: Developers work in the dev workspace via Databricks Repos (linked to Azure DevOps Git). They commit changes, push to a feature branch, and create a PR. The CI pipeline runs on the PR: validates the DABs bundle, lints Python code, and runs unit tests. After PR approval and merge to main, the CD pipeline deploys via DABs: first to dev (auto-deploy), then to staging (approval gate), then to production (2 approvals, main branch only). This workspace-first, Git-backed approach preserves developer productivity while enforcing production safety.

Q: How do DABs targets enable multi-environment deployment? A: The databricks.yml file defines shared resources (jobs, pipelines, notebooks) and environment-specific targets (dev, staging, prod). Each target specifies a workspace URL, catalog name, and resource overrides (cluster size, schedule). The deploy command databricks bundle deploy --target prod applies the prod overrides. Same code, same YAML, different environments. Dev targets typically override schedule to null and use smaller clusters; prod targets use full-size clusters and production catalogs.

Q: How do you test Databricks notebooks in a CI/CD pipeline? A: Two levels of testing. Unit tests run on the CI agent without a Databricks cluster: extract transformation logic into importable Python functions and test with pytest and a local PySpark session. Integration tests run on a Databricks cluster: deploy to staging, run the job with databricks bundle run, then validate results (row counts, schema checks, duplicate checks). Unit tests run on every PR (fast, free). Integration tests run after staging deployment (slower, requires cluster).

Q: How do you handle Unity Catalog objects across environments? A: Use separate catalogs per environment (dev_catalog, staging_catalog, prod_catalog) within a shared metastore. Parameterize the catalog name in databricks.yml as a variable. Each DABs target sets the correct catalog. Notebooks receive the catalog name as a widget parameter and run USE CATALOG at the start. This ensures dev notebooks never touch prod data, and the same notebook code works across all environments by changing only the catalog parameter.

Q: What is the difference between deploying via Repos sync vs DABs deploy? A: Repos sync updates the Git content visible in the workspace but does not create or update jobs, clusters, or pipelines. It is a code sync, not a deployment. DABs deploy creates and updates the full set of Databricks resources defined in databricks.yml: jobs, pipelines, cluster configs, permissions, and notebook paths. For production CI/CD, always use DABs deploy because it manages the complete lifecycle of Databricks resources, not just the code files.

Q: What approval gates should you configure for Databricks deployments? A: Configure Azure DevOps environments with escalating approvals: dev has no approvals (auto-deploy for fast iteration), staging requires one approver (tech lead validates), and production requires two approvers (tech lead plus engineering manager). Additionally, add a branch control check on production to allow deployments only from the main branch. For critical pipelines, add a manual validation step after staging integration tests pass and before production deployment begins.

Wrapping Up

CI/CD for Databricks with Azure DevOps combines three tools: Azure DevOps for pipelines and Git, DABs for defining Databricks resources as code, and service principals for secure authentication. The workflow is workspace-first for development, Git-backed for deployment: developers iterate fast in the dev workspace, changes flow through PRs with CI validation, and DABs deploy through environments with approval gates.

The key principle: no one touches production directly. Every change goes through code → PR → CI → staging → approval → production. This is not overhead — it is the safety net that prevents a notebook typo from breaking a production pipeline at 2 AM.

Related posts: – Databricks Asset Bundles (DABs) – Databricks Git Integration & CI/CD – Terraform for Data Engineers – YAML Pipelines Deep Dive – Azure DevOps Overview

Related Posts

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top