YAML Pipelines Deep Dive for Data Engineers: Triggers, Variables, Parameters, Conditions, Templates, Multi-Stage Deployments, Approval Gates, Matrix Strategy, Each Loops, and Production Pipeline Patterns

Leave a Comment / Azure, Data Engineering, DevOps

Table of Contents

In the previous post, we covered the Azure DevOps platform, its five services, and a basic YAML pipeline. This post goes deep into YAML pipelines — the engine that automates everything in modern data engineering. We will cover every building block: triggers, variables, parameters, conditions, templates, multi-stage deployments, approval gates, matrix strategies, each loops, and real-world data engineering pipeline examples you can copy and adapt.

Analogy — A recipe book with smart features. A YAML pipeline is like a recipe book that cooks itself. Triggers are the dinner bell — they tell the kitchen when to start cooking (on every commit, on a schedule, or when someone presses a button). Variables are the ingredient labels — “use butter for dev, use margarine for prod” (same recipe, different inputs). Parameters are the menu choices — the diner picks “spicy” or “mild” before the kitchen starts. Conditions are dietary restrictions — “skip the dessert stage if the diner is diabetic.” Templates are the base recipes — write “make pasta” once, and every dish that includes pasta references the same template. And stages are the courses — appetizer (build), main course (deploy to dev), dessert (deploy to prod) — served in order, with the waiter (approval gate) checking before each course.

YAML Pipeline Anatomy Revisited

# Every YAML pipeline has this structure:
trigger: ...          # WHEN does the pipeline run?
pr: ...               # Run on pull requests?
pool: ...             # WHERE does it run? (which agent)
variables: ...        # WHAT configuration values?
parameters: ...       # WHAT user inputs (compile-time)?
stages:               # The main work (or just steps: for simple pipelines)
  - stage: Build
    jobs:
      - job: Test
        steps:
          - script: echo "Hello"
Hierarchy:

  Pipeline
    ├── Stage (logical grouping -- Build, Deploy Dev, Deploy Prod)
    |     ├── Job (runs on ONE agent -- can have multiple jobs per stage)
    |     |     ├── Step: script (run a bash/PowerShell command)
    |     |     ├── Step: task (run a built-in Azure DevOps task)
    |     |     └── Step: template (reference a reusable step template)
    |     └── Job: (another job, runs in parallel by default)
    └── Stage: (another stage, runs after the previous by default)

  Simple pipelines can skip stages and jobs:
    steps:          # Just steps -- Azure DevOps wraps them in a default stage/job
      - script: echo "Simple pipeline"

Triggers — When Pipelines Run

Push Triggers

# Trigger on push to specific branches
trigger:
  branches:
    include:
      - main
      - release/*        # All release branches
    exclude:
      - feature/experimental/*
  paths:
    include:
      - terraform/**     # Only trigger when Terraform files change
      - databricks/**    # Or Databricks files
    exclude:
      - docs/**          # Never trigger for documentation changes
      - '*.md'           # Never trigger for markdown files

Pull Request Triggers

# Run CI on every pull request targeting main
pr:
  branches:
    include:
      - main
      - develop
  paths:
    include:
      - src/**
      - tests/**
  drafts: false          # Don't run on draft PRs

Scheduled Triggers

# Nightly build at 2 AM UTC
schedules:
  - cron: '0 2 * * *'
    displayName: 'Nightly validation'
    branches:
      include:
        - main
    always: true          # Run even if no code changes

  # Weekly infrastructure validation (Sundays at midnight)
  - cron: '0 0 * * 0'
    displayName: 'Weekly Terraform plan'
    branches:
      include:
        - main
    always: true

Manual Trigger

# No automatic triggers -- manual only
trigger: none
pr: none

# Pipeline can only be run manually from Azure DevOps UI
# or via REST API / az pipelines run

Resource Triggers (Pipeline Chaining)

# Trigger this pipeline when another pipeline completes
resources:
  pipelines:
    - pipeline: ci-build          # Alias for the source pipeline
      source: 'CI-Build-Pipeline' # Name of the source pipeline
      trigger:
        branches:
          include:
            - main

# This pipeline runs AFTER CI-Build-Pipeline succeeds on main
# Use case: CI pipeline builds → CD pipeline deploys

Variables — Configuration at Every Level

Variables make pipelines flexible. They can be defined at the pipeline level, stage level, job level, in variable groups, or from Key Vault.

Variable Scopes

# Pipeline-level variables (available everywhere)
variables:
  environment: 'dev'
  pythonVersion: '3.11'
  terraformVersion: '1.7.0'

stages:
  - stage: Build
    # Stage-level variables (override pipeline-level)
    variables:
      buildConfig: 'Release'
    jobs:
      - job: Test
        # Job-level variables (most specific)
        variables:
          testFilter: 'unit'
        steps:
          - script: |
              echo "Env: $(environment)"        # dev
              echo "Config: $(buildConfig)"     # Release
              echo "Filter: $(testFilter)"      # unit

Variable Groups

# Reference variable groups (defined in Pipelines > Library)
variables:
  - group: 'dev-variables'         # Contains STORAGE_ACCOUNT, DATABRICKS_HOST
  - group: 'dev-secrets'           # Linked to Key Vault
  - name: customVar                # Inline variable alongside groups
    value: 'my-value'

steps:
  - script: |
      echo "Storage: $(STORAGE_ACCOUNT)"     # From variable group
      echo "Token: $(DATABRICKS_TOKEN)"      # From Key Vault (masked in logs)
      echo "Custom: $(customVar)"            # Inline

Runtime vs Compile-Time Variables

Two types of variables:

  Runtime variables: $(variableName)
    - Resolved when the pipeline RUNS
    - Can be set by scripts during execution
    - Can come from variable groups / Key Vault
    - Use for: secrets, environment-specific values

  Compile-time expressions: ${{ variables.variableName }}
    - Resolved when the pipeline is COMPILED (before it runs)
    - Must be known at parse time
    - Use for: template expressions, conditional insertion of stages/jobs

  Macro syntax: $(variableName)
    - Resolved just before each step runs
    - Most common syntax for referencing variables in scripts

  Example:
    ${{ if eq(variables.environment, 'prod') }}:  # Compile-time decision
      - stage: DeployProd                          # This stage is included or excluded at compile time

    - script: echo "$(environment)"               # Runtime value in a script

Parameters — Compile-Time Inputs

Parameters let users provide inputs when they manually trigger a pipeline. Unlike variables, parameters have types (string, number, boolean, object) and can have allowed values.

parameters:
  - name: environment
    displayName: 'Target Environment'
    type: string
    default: 'dev'
    values:
      - dev
      - staging
      - prod

  - name: runTests
    displayName: 'Run Tests?'
    type: boolean
    default: true

  - name: terraformAction
    displayName: 'Terraform Action'
    type: string
    default: 'plan'
    values:
      - plan
      - apply
      - destroy

  - name: regions
    displayName: 'Deploy Regions'
    type: object
    default:
      - eastus
      - westus2

trigger: none    # Manual only -- user selects parameters in UI

stages:
  - stage: Deploy
    displayName: 'Deploy to ${{ parameters.environment }}'
    jobs:
      - job: TerraformDeploy
        steps:
          - script: |
              echo "Environment: ${{ parameters.environment }}"
              echo "Action: ${{ parameters.terraformAction }}"
            displayName: 'Show parameters'

          - ${{ if eq(parameters.runTests, true) }}:
            - script: pytest tests/ -v
              displayName: 'Run tests (enabled)'
Parameters vs Variables:

  Parameters:
    - Defined at compile time (before pipeline runs)
    - Have types (string, boolean, number, object)
    - Can have restricted allowed values
    - Show as input fields when manually triggering
    - Referenced with: ${{ parameters.name }}

  Variables:
    - Can be set at runtime (by scripts, variable groups)
    - Always strings
    - No type validation
    - Referenced with: $(name) or ${{ variables.name }}

  Rule of thumb:
    Use parameters for user choices (environment, action, flags)
    Use variables for configuration values (URLs, paths, versions)

Conditions — Controlling What Runs

Conditions control whether a stage, job, or step executes based on pipeline state, variables, or previous stage results.

Stage and Job Conditions

stages:
  - stage: Build
    jobs:
      - job: Test
        steps:
          - script: pytest tests/ -v

  - stage: DeployDev
    dependsOn: Build
    condition: succeeded()             # Run only if Build succeeded (default)

  - stage: DeployProd
    dependsOn: DeployDev
    condition: |
      and(
        succeeded(),
        eq(variables['Build.SourceBranch'], 'refs/heads/main')
      )
    # Run only if DeployDev succeeded AND we are on the main branch

Step Conditions

steps:
  - script: echo "Always runs"

  - script: echo "Only on main"
    condition: eq(variables['Build.SourceBranch'], 'refs/heads/main')

  - script: echo "Only on PRs"
    condition: eq(variables['Build.Reason'], 'PullRequest')

  - script: echo "Previous step failed"
    condition: failed()

  - script: echo "Runs regardless of previous results"
    condition: always()

  - script: echo "Only if variable is set"
    condition: ne(variables['DEPLOY_FLAG'], '')

Compile-Time Conditions (if/else)

parameters:
  - name: environment
    type: string
    default: 'dev'

stages:
  # Always include dev
  - stage: DeployDev
    displayName: 'Deploy to Dev'
    jobs:
      - job: Deploy
        steps:
          - script: echo "Deploying to dev"

  # Only include prod stage when parameter is 'prod'
  - ${{ if eq(parameters.environment, 'prod') }}:
    - stage: DeployProd
      displayName: 'Deploy to Production'
      dependsOn: DeployDev
      jobs:
        - job: Deploy
          steps:
            - script: echo "Deploying to prod"

Templates — Reusable Pipeline Components

Templates let you define reusable pipeline components — steps, jobs, stages, or entire pipelines — and reference them from multiple pipelines.

Analogy — LEGO instruction booklets. Each template is a sub-booklet for building a specific component (wheels, engine, cockpit). The main booklet (your pipeline) references these sub-booklets instead of repeating the same instructions. Change the wheel design in one sub-booklet, and every vehicle that uses it gets the update automatically.

Step Template

# templates/install-python.yml
parameters:
  - name: version
    type: string
    default: '3.11'

steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '${{ parameters.version }}'
    displayName: 'Set up Python ${{ parameters.version }}'

  - script: |
      pip install --upgrade pip
      pip install -r requirements.txt
    displayName: 'Install dependencies'
# Main pipeline -- references the template
stages:
  - stage: Build
    jobs:
      - job: Test
        steps:
          - template: templates/install-python.yml   # Reuse!
            parameters:
              version: '3.11'
          - script: pytest tests/ -v
            displayName: 'Run tests'

Job Template

# templates/terraform-job.yml
parameters:
  - name: environment
    type: string
  - name: serviceConnection
    type: string
  - name: action
    type: string
    default: 'plan'

jobs:
  - job: Terraform_${{ parameters.environment }}
    displayName: 'Terraform ${{ parameters.action }} (${{ parameters.environment }})'
    pool:
      vmImage: 'ubuntu-latest'
    steps:
      - script: |
          terraform init -backend-config="environments/${{ parameters.environment }}/backend.conf"
          terraform ${{ parameters.action }} -var-file="environments/${{ parameters.environment }}/terraform.tfvars"
        displayName: 'Terraform ${{ parameters.action }}'
        env:
          ARM_CLIENT_ID: $(ARM_CLIENT_ID)
          ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
          ARM_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
          ARM_TENANT_ID: $(ARM_TENANT_ID)
# Main pipeline -- uses the job template for each environment
stages:
  - stage: PlanDev
    jobs:
      - template: templates/terraform-job.yml
        parameters:
          environment: 'dev'
          serviceConnection: 'azure-dev'
          action: 'plan'

  - stage: ApplyDev
    dependsOn: PlanDev
    jobs:
      - template: templates/terraform-job.yml
        parameters:
          environment: 'dev'
          serviceConnection: 'azure-dev'
          action: 'apply'

Stage Template

# templates/deploy-stage.yml
parameters:
  - name: environment
    type: string
  - name: serviceConnection
    type: string
  - name: variableGroup
    type: string
  - name: dependsOn
    type: string
    default: ''

stages:
  - stage: Deploy_${{ parameters.environment }}
    displayName: 'Deploy to ${{ parameters.environment }}'
    ${{ if ne(parameters.dependsOn, '') }}:
      dependsOn: ${{ parameters.dependsOn }}
    variables:
      - group: ${{ parameters.variableGroup }}
    jobs:
      - deployment: DeployJob
        environment: ${{ parameters.environment }}
        pool:
          vmImage: 'ubuntu-latest'
        strategy:
          runOnce:
            deploy:
              steps:
                - task: AzureCLI@2
                  inputs:
                    azureSubscription: ${{ parameters.serviceConnection }}
                    scriptType: 'bash'
                    scriptLocation: 'inlineScript'
                    inlineScript: |
                      echo "Deploying to ${{ parameters.environment }}"
                      echo "Storage: $(STORAGE_ACCOUNT)"
                      echo "Databricks: $(DATABRICKS_HOST)"
# Main pipeline -- clean and DRY
trigger:
  - main

stages:
  - stage: Build
    jobs:
      - job: Test
        steps:
          - script: pytest tests/ -v

  - template: templates/deploy-stage.yml
    parameters:
      environment: 'dev'
      serviceConnection: 'azure-dev'
      variableGroup: 'dev-variables'
      dependsOn: 'Build'

  - template: templates/deploy-stage.yml
    parameters:
      environment: 'staging'
      serviceConnection: 'azure-staging'
      variableGroup: 'staging-variables'
      dependsOn: 'Deploy_dev'

  - template: templates/deploy-stage.yml
    parameters:
      environment: 'prod'
      serviceConnection: 'azure-prod'
      variableGroup: 'prod-variables'
      dependsOn: 'Deploy_staging'

Multi-Stage Pipelines — Dev to Prod in One File

A multi-stage pipeline handles the entire lifecycle: build, test, deploy to dev, deploy to staging, deploy to production — all in one YAML file with gates between stages.

# Complete multi-stage pipeline for a Databricks project
trigger:
  branches:
    include:
      - main
  paths:
    include:
      - databricks/**

variables:
  - name: pythonVersion
    value: '3.11'

stages:
  # Stage 1: Build and Test
  - stage: Build
    displayName: 'Build & Test'
    pool:
      vmImage: 'ubuntu-latest'
    jobs:
      - job: UnitTests
        steps:
          - task: UsePythonVersion@0
            inputs:
              versionSpec: '$(pythonVersion)'
          - script: |
              pip install -r requirements.txt
              pytest tests/unit/ -v --junitxml=test-results.xml
            displayName: 'Run unit tests'
          - task: PublishTestResults@2
            inputs:
              testResultsFiles: 'test-results.xml'
            displayName: 'Publish test results'

  # Stage 2: Deploy to Dev
  - stage: DeployDev
    displayName: 'Deploy to Dev'
    dependsOn: Build
    condition: succeeded()
    variables:
      - group: 'dev-variables'
    jobs:
      - deployment: DeployDatabricksDev
        environment: 'dev'
        pool:
          vmImage: 'ubuntu-latest'
        strategy:
          runOnce:
            deploy:
              steps:
                - script: |
                    pip install databricks-cli
                    databricks bundle deploy --target dev
                  displayName: 'Deploy DABs to Dev'
                  env:
                    DATABRICKS_HOST: $(DATABRICKS_HOST)
                    DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)

  # Stage 3: Deploy to Staging (requires approval)
  - stage: DeployStaging
    displayName: 'Deploy to Staging'
    dependsOn: DeployDev
    condition: succeeded()
    variables:
      - group: 'staging-variables'
    jobs:
      - deployment: DeployDatabricksStaging
        environment: 'staging'         # Configure approval on this environment
        pool:
          vmImage: 'ubuntu-latest'
        strategy:
          runOnce:
            deploy:
              steps:
                - script: |
                    databricks bundle deploy --target staging
                  displayName: 'Deploy DABs to Staging'
                  env:
                    DATABRICKS_HOST: $(DATABRICKS_HOST)
                    DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)

  # Stage 4: Deploy to Production (requires 2 approvals)
  - stage: DeployProd
    displayName: 'Deploy to Production'
    dependsOn: DeployStaging
    condition: |
      and(
        succeeded(),
        eq(variables['Build.SourceBranch'], 'refs/heads/main')
      )
    variables:
      - group: 'prod-variables'
    jobs:
      - deployment: DeployDatabricksProd
        environment: 'production'      # Configure 2 approvers on this environment
        pool:
          vmImage: 'ubuntu-latest'
        strategy:
          runOnce:
            deploy:
              steps:
                - script: |
                    databricks bundle deploy --target prod
                  displayName: 'Deploy DABs to Production'
                  env:
                    DATABRICKS_HOST: $(DATABRICKS_HOST)
                    DATABRICKS_TOKEN: $(DATABRICKS_TOKEN)

Environments and Approval Gates

Environments are logical deployment targets that track deployment history and enforce approval policies.

Setting up environments with approvals:

  1. Pipelines > Environments > New Environment
     Name: "dev" → No approvals (auto-deploy)

  2. Pipelines > Environments > New Environment
     Name: "staging" → 1 approver
       Click staging > Approvals and Checks > Add > Approvals
       Add 1 reviewer (tech lead)

  3. Pipelines > Environments > New Environment
     Name: "production" → 2 approvers + branch check
       Click production > Approvals and Checks > Add:
       - Approvals: 2 reviewers (tech lead + manager)
       - Branch control: only allow deployments from refs/heads/main
       - Business hours: only deploy Mon-Fri 9 AM - 5 PM (optional)

  How it works:
    Pipeline reaches DeployProd stage
    → Azure DevOps pauses the pipeline
    → Sends notification to approvers
    → Approvers review and approve/reject in the UI
    → Pipeline continues or stops

  Deployment history:
    Each environment shows a full history:
    - What was deployed
    - When it was deployed
    - Who approved
    - Which pipeline run

Matrix Strategy — Parallel Execution

Matrix strategy runs the same job multiple times with different variable combinations — useful for testing across Python versions, platforms, or environments.

jobs:
  - job: Test
    strategy:
      matrix:
        Python311:
          pythonVersion: '3.11'
        Python312:
          pythonVersion: '3.12'
        Python313:
          pythonVersion: '3.13'
      maxParallel: 3           # Run all 3 in parallel

    steps:
      - task: UsePythonVersion@0
        inputs:
          versionSpec: '$(pythonVersion)'
      - script: |
          pip install -r requirements.txt
          pytest tests/ -v
        displayName: 'Test on Python $(pythonVersion)'
# Matrix for multi-environment validation
jobs:
  - job: ValidateInfra
    strategy:
      matrix:
        Dev:
          env: 'dev'
          tfvarsFile: 'dev.tfvars'
        Staging:
          env: 'staging'
          tfvarsFile: 'staging.tfvars'
        Prod:
          env: 'prod'
          tfvarsFile: 'prod.tfvars'
    steps:
      - script: |
          terraform init -backend-config="$(env)-backend.conf"
          terraform plan -var-file="$(tfvarsFile)"
        displayName: 'Terraform plan for $(env)'

Each Loops — Dynamic Stage Generation

The ${{ each }} expression dynamically generates stages, jobs, or steps from a parameter list — powerful for multi-environment deployments from a single pipeline.

parameters:
  - name: environments
    type: object
    default:
      - name: dev
        serviceConnection: azure-dev
        variableGroup: dev-variables
        approvals: false
      - name: staging
        serviceConnection: azure-staging
        variableGroup: staging-variables
        approvals: true
      - name: prod
        serviceConnection: azure-prod
        variableGroup: prod-variables
        approvals: true

trigger:
  - main

stages:
  - stage: Build
    jobs:
      - job: Test
        steps:
          - script: pytest tests/ -v

  # Dynamically generate a deployment stage for EACH environment
  - ${{ each env in parameters.environments }}:
    - stage: Deploy_${{ env.name }}
      displayName: 'Deploy to ${{ env.name }}'
      ${{ if eq(env.name, 'dev') }}:
        dependsOn: Build
      ${{ if eq(env.name, 'staging') }}:
        dependsOn: Deploy_dev
      ${{ if eq(env.name, 'prod') }}:
        dependsOn: Deploy_staging
      variables:
        - group: ${{ env.variableGroup }}
      jobs:
        - deployment: Deploy
          environment: ${{ env.name }}
          pool:
            vmImage: 'ubuntu-latest'
          strategy:
            runOnce:
              deploy:
                steps:
                  - task: AzureCLI@2
                    inputs:
                      azureSubscription: ${{ env.serviceConnection }}
                      scriptType: 'bash'
                      scriptLocation: 'inlineScript'
                      inlineScript: |
                        echo "Deploying to ${{ env.name }}"

Pipeline Artifacts — Passing Data Between Stages

Stages run on different agents, so they do not share files. Use pipeline artifacts to pass data between stages.

stages:
  - stage: Build
    jobs:
      - job: BuildArtifact
        steps:
          - script: |
              mkdir output
              terraform plan -out=output/plan.tfplan
              cp terraform.tfstate output/
            displayName: 'Generate Terraform plan'

          - publish: output/
            artifact: terraform-plan
            displayName: 'Publish plan artifact'

  - stage: Apply
    dependsOn: Build
    jobs:
      - job: ApplyPlan
        steps:
          - download: current
            artifact: terraform-plan
            displayName: 'Download plan artifact'

          - script: |
              terraform apply $(Pipeline.Workspace)/terraform-plan/plan.tfplan
            displayName: 'Apply saved plan'

Real-World Data Engineering Pipelines

CI Pipeline for Databricks Notebooks

# ci-databricks.yml: Runs on every PR targeting main
trigger: none
pr:
  branches:
    include:
      - main
  paths:
    include:
      - databricks/**

pool:
  vmImage: 'ubuntu-latest'

steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.11'

  - script: |
      pip install pytest pyspark databricks-sdk
      pytest databricks/tests/ -v --junitxml=results.xml
    displayName: 'Run notebook unit tests'

  - task: PublishTestResults@2
    inputs:
      testResultsFiles: 'results.xml'
    displayName: 'Publish test results'

  - script: |
      pip install ruff
      ruff check databricks/ --output-format=github
    displayName: 'Lint Python code'

CD Pipeline for Terraform Infrastructure

# cd-terraform.yml: Deploys infrastructure on merge to main
trigger:
  branches:
    include:
      - main
  paths:
    include:
      - terraform/**

parameters:
  - name: action
    displayName: 'Terraform Action'
    type: string
    default: 'plan'
    values:
      - plan
      - apply

stages:
  - stage: Plan
    displayName: 'Terraform Plan'
    jobs:
      - job: TerraformPlan
        pool:
          vmImage: 'ubuntu-latest'
        variables:
          - group: 'terraform-variables'
        steps:
          - script: |
              curl -fsSL https://releases.hashicorp.com/terraform/1.7.0/terraform_1.7.0_linux_amd64.zip -o tf.zip
              unzip tf.zip && sudo mv terraform /usr/local/bin/
              terraform --version
            displayName: 'Install Terraform'

          - script: |
              cd terraform/
              terraform init \
                -backend-config="resource_group_name=$(TF_BACKEND_RG)" \
                -backend-config="storage_account_name=$(TF_BACKEND_STORAGE)" \
                -backend-config="container_name=tfstate" \
                -backend-config="key=dataplatform.tfstate"
              terraform plan -var-file="environments/prod.tfvars" -out=plan.tfplan
            displayName: 'Terraform Plan'
            env:
              ARM_CLIENT_ID: $(ARM_CLIENT_ID)
              ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
              ARM_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
              ARM_TENANT_ID: $(ARM_TENANT_ID)

          - publish: terraform/plan.tfplan
            artifact: tf-plan
            condition: eq('${{ parameters.action }}', 'apply')

  - ${{ if eq(parameters.action, 'apply') }}:
    - stage: Apply
      displayName: 'Terraform Apply'
      dependsOn: Plan
      jobs:
        - deployment: TerraformApply
          environment: 'production'
          pool:
            vmImage: 'ubuntu-latest'
          variables:
            - group: 'terraform-variables'
          strategy:
            runOnce:
              deploy:
                steps:
                  - download: current
                    artifact: tf-plan
                  - script: |
                      cd terraform/
                      terraform init -backend-config="..."
                      terraform apply $(Pipeline.Workspace)/tf-plan/plan.tfplan
                    displayName: 'Terraform Apply'
                    env:
                      ARM_CLIENT_ID: $(ARM_CLIENT_ID)
                      ARM_CLIENT_SECRET: $(ARM_CLIENT_SECRET)
                      ARM_SUBSCRIPTION_ID: $(ARM_SUBSCRIPTION_ID)
                      ARM_TENANT_ID: $(ARM_TENANT_ID)

Common Mistakes

  1. Using runtime variables in compile-time expressions. Writing ${{ if eq($(myVar), 'prod') }} fails because $(myVar) is resolved at runtime, but ${{ }} expressions are resolved at compile time. Use ${{ if eq(variables.myVar, 'prod') }} for compile-time or condition: eq(variables['myVar'], 'prod') for runtime conditions.

  2. Not using templates for repeated patterns. Copy-pasting the same deploy steps across three environment stages creates maintenance nightmares. When the deploy process changes, you must update it in three places. Extract repeated patterns into templates and reference them with different parameters.

  3. Hardcoding secrets in YAML files. Writing DATABRICKS_TOKEN: 'dapi123...' in your pipeline YAML is a security disaster — it is visible to anyone with repo access and stored in Git history forever. Use variable groups linked to Key Vault and reference secrets with $(SECRET_NAME).

  4. Not setting dependsOn between stages. Without explicit dependsOn, stages run sequentially in order. But if you add a new stage between existing ones, the order might not be what you expect. Always use dependsOn to make dependencies explicit, especially in multi-stage pipelines.

  5. Forgetting to configure approval gates on production environments. Without approvals, a merge to main automatically deploys to production. Create an environment called “production” in Azure DevOps and add approval checks before using it in deployment jobs.

  6. Using condition: always() on deployment steps. If a previous step fails, always() makes the deployment step run anyway, potentially deploying broken code. Use condition: succeeded() (the default) for deployment steps. Reserve always() for cleanup steps like deleting temporary resources.

  7. Not publishing test results. Running pytest without PublishTestResults@2 means test failures only appear in build logs. Published results show up in the Tests tab with trends, making it easy to track which tests fail and when they started failing.

  8. Creating one monolithic pipeline instead of separating CI and CD. A single pipeline that builds, tests, and deploys to all environments is hard to debug and slow to re-run. Separate CI (runs on every PR, fast feedback) from CD (runs on merge to main, multi-stage deployment). Use resource triggers to chain them.

Interview Questions

Q: What is the difference between triggers and PR triggers in YAML pipelines? A: The trigger section defines push triggers — the pipeline runs when code is pushed to matching branches. The pr section defines pull request triggers — the pipeline runs when a PR is created or updated targeting matching branches. Both support branch and path filters. For data engineering, use push triggers on main for CD pipelines (deploy on merge) and PR triggers for CI pipelines (test before merge). Setting trigger: none and pr: none makes the pipeline manual-only.

Q: What is the difference between parameters and variables? A: Parameters are compile-time inputs with types (string, boolean, number, object) and allowed values. They are resolved before the pipeline runs and can control which stages or jobs are included. Variables are runtime values (always strings) that can be set by scripts, variable groups, or Key Vault. Use parameters for user choices (which environment to deploy to) and variables for configuration values (connection strings, resource names). Parameters use ${{ parameters.name }} syntax; variables use $(name) syntax.

Q: How do templates work and why are they important? A: Templates are reusable YAML files that define steps, jobs, or stages. They accept parameters for customization. The main pipeline references templates using - template: path/to/template.yml with parameters. Templates are important because they enforce consistency (every deployment follows the same pattern), reduce duplication (write once, reference everywhere), and simplify maintenance (update the template, all pipelines get the update). For data engineering, common templates include Terraform plan/apply, Databricks DABs deploy, and Python test/lint.

Q: What are environments and how do approval gates work? A: Environments are logical deployment targets (dev, staging, production) defined in Azure DevOps. They provide deployment history tracking, approval policies, and branch restrictions. When a deployment job targets an environment with approvals, the pipeline pauses and sends notifications to configured approvers. Approvers review the deployment in the Azure DevOps UI and approve or reject. Branch control ensures only the main branch can deploy to production. Business hours checks prevent deployments outside work hours.

Q: How does the each loop work for multi-environment deployments? A: The ${{ each }} expression iterates over a parameter list at compile time, generating stages, jobs, or steps dynamically. You define environments as a parameter of type object with properties (name, service connection, variable group). The each loop generates a deployment stage for each environment from a single block of YAML. This eliminates copy-paste and ensures every environment follows the same deployment pattern. Changes to the deployment logic are made in one place.

Q: How do you pass data between stages in Azure Pipelines? A: Stages run on different agents, so they do not share file systems. Use the publish step to upload files as pipeline artifacts and the download step to retrieve them in subsequent stages. For example, a Build stage publishes a Terraform plan file, and an Apply stage downloads and applies it. This ensures the exact plan that was reviewed is the one that gets applied. Output variables can also pass small values between jobs using isOutput=true and the stageDependencies syntax.

Q: What is the recommended pipeline structure for a data engineering project? A: Separate CI and CD pipelines. The CI pipeline runs on every pull request (PR trigger): it runs unit tests, lints code, and validates Terraform plans. The CD pipeline runs on merge to main (push trigger): it deploys through dev, staging, and production stages with approval gates. Use templates for shared deployment logic. Use variable groups per environment linked to Key Vault. Use environments with approval policies on staging and production. Chain CI and CD using resource triggers if they are separate pipelines.

Wrapping Up

YAML pipelines are the backbone of data engineering automation. Triggers control when pipelines run. Variables and parameters make them flexible. Conditions control which parts execute. Templates eliminate duplication. Multi-stage pipelines with approval gates enforce the dev-to-prod promotion path. And each loops generate dynamic, DRY pipelines that handle any number of environments from a single definition.

In the next post, we will take everything we have learned and apply it to Terraform — writing infrastructure as code that provisions Databricks workspaces, storage accounts, Key Vaults, and entire data platforms, deployed through the YAML pipelines we just built.

Related posts: – Azure DevOps Overview – CI/CD for ADF with Azure DevOps – Databricks Asset Bundles (DABs) – Databricks Git Integration & CI/CD – Fabric Git Integration & Deployment Pipelines

Related Posts

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top