Azure DevOps for Data Engineers: Repos, Pipelines, Boards, Artifacts, Service Connections, Variable Groups, YAML Pipelines, Branching Strategies, and Setting Up Your Data Platform Project

Table of Contents

• What Is Azure DevOps?
• The Five Core Services
• Azure DevOps Organizations and Projects
• Azure Repos — Git-Based Source Control
• Branching Strategies for Data Engineering
• Pull Requests and Code Reviews
• Azure Pipelines — CI/CD Automation
• YAML Pipelines — The Foundation
• Stages, Jobs, and Steps — Pipeline Anatomy
• Service Connections — Connecting to Azure Resources
• Variable Groups and Secrets
• Azure Boards — Work Tracking for Data Teams
• Azure Artifacts — Package Management
• Azure DevOps vs GitHub — Which to Choose
• Setting Up Azure DevOps for a Data Engineering Project
• Common Mistakes
• Interview Questions
• Wrapping Up

Our existing CI/CD posts cover ADF and Synapse CI/CD with GitHub, ADF CI/CD with Azure DevOps, Databricks Git Integration, and Databricks Asset Bundles. But those posts assume you already know Azure DevOps. This post is the foundation — a complete walkthrough of the Azure DevOps platform from a data engineer’s perspective: what each service does, how they connect, and how to set up a project for data engineering work.

Analogy — A construction company headquarters. Azure DevOps is the headquarters of a construction company. Repos is the blueprint vault — every blueprint (code) is version-controlled, and architects (engineers) collaborate through reviews before a blueprint is approved. Pipelines is the construction crew — when a blueprint is finalized, the crew automatically builds, inspects, and delivers the building (deploys the code). Boards is the project management office — tracking which buildings are planned, in progress, and completed. Artifacts is the materials warehouse — storing reusable components (packages, libraries) that multiple construction projects share. And Test Plans is the inspection department — verifying that buildings meet quality standards before occupants move in.

What Is Azure DevOps?

Azure DevOps is Microsoft’s integrated platform for software development lifecycle management. It provides source control, CI/CD pipelines, work tracking, package management, and testing — all in one place, tightly integrated with Azure and the Microsoft ecosystem.

Why data engineers need Azure DevOps:

  Without DevOps:
    - Code lives on local machines or shared drives
    - Deployments are manual (copy-paste between environments)
    - No audit trail of who changed what and when
    - "It works on my laptop" syndrome
    - One person's mistake breaks production with no rollback

  With Azure DevOps:
    - Code is version-controlled (every change tracked, reviewable, reversible)
    - Deployments are automated (push to main → tests run → deploy to prod)
    - Full audit trail (who approved, who merged, who deployed, when)
    - Consistent environments (dev, staging, prod are identical)
    - Instant rollback (revert to the previous deployment in minutes)

  For data engineers specifically:
    - Version-control ADF/Synapse pipelines, Databricks notebooks, Fabric items
    - Automate deployment of Terraform infrastructure (Databricks workspaces, storage accounts)
    - Run data quality tests before promoting to production
    - Track data engineering work items (pipeline bugs, new data sources, schema changes)

The Five Core Services

Azure DevOps consists of five services that can be used independently or together:

You can adopt services incrementally:

  Stage 1: Just Repos (version control your code)
  Stage 2: Repos + Pipelines (automate deployments)
  Stage 3: Repos + Pipelines + Boards (track work + automate)
  Stage 4: All five (full DevOps maturity)

  Most data engineering teams start at Stage 2 and grow from there.

Azure DevOps Organizations and Projects

Azure DevOps hierarchy:

  Organization (company-level)
    |
    ├── Project: Data Platform
    |     ├── Repos: adf-pipelines, databricks-notebooks, terraform-infra
    |     ├── Pipelines: ci-adf, cd-databricks, deploy-terraform
    |     ├── Boards: Sprint 1, Sprint 2, Backlog
    |     └── Artifacts: shared-python-utils
    |
    ├── Project: Analytics Platform
    |     ├── Repos: power-bi-models, dbt-transformations
    |     ├── Pipelines: ci-dbt, deploy-power-bi
    |     └── Boards: Analytics backlog
    |
    └── Settings (organization-level policies, billing, users)

  Organization = your company (one per company typically)
  Project = a team or platform (one per major initiative)
  Repos = individual code repositories within a project

  URL pattern: https://dev.azure.com/{organization}/{project}

Setting up an organization:

  1. Go to https://dev.azure.com
  2. Sign in with your Microsoft/Azure AD account
  3. Create Organization: "MyCompany" (or your company name)
  4. Create Project: "DataPlatform"
     - Visibility: Private (enterprise) or Public (open source)
     - Version control: Git (always use Git, not TFVC)
     - Work item process: Agile (most common for data teams)

Azure Repos — Git-Based Source Control

Azure Repos is a Git hosting service — functionally similar to GitHub or GitLab, but integrated with Azure DevOps pipelines, boards, and artifacts.

Key Repos concepts:

  Repository: a Git repo within a project (you can have multiple)
  Branch: a parallel version of code (main, develop, feature/add-new-source)
  Commit: a saved snapshot of changes with a message
  Pull Request (PR): a request to merge changes from one branch to another
  Branch Policy: rules that must pass before a PR can be merged

  For data engineers, a typical repo structure:

  data-platform/
    ├── terraform/              # Infrastructure as Code
    |     ├── main.tf
    |     ├── variables.tf
    |     ├── environments/
    |     |     ├── dev.tfvars
    |     |     ├── staging.tfvars
    |     |     └── prod.tfvars
    |     └── modules/
    |           ├── databricks/
    |           ├── storage/
    |           └── keyvault/
    |
    ├── databricks/             # Databricks notebooks and configs
    |     ├── notebooks/
    |     ├── dabs/
    |     |     ├── databricks.yml
    |     |     └── resources/
    |     └── tests/
    |
    ├── adf/                    # ADF pipeline definitions
    |     └── (managed by ADF Git integration)
    |
    ├── pipelines/              # Azure DevOps YAML pipelines
    |     ├── ci-terraform.yml
    |     ├── cd-databricks.yml
    |     └── cd-adf.yml
    |
    ├── scripts/                # Utility scripts
    |     ├── deploy.sh
    |     └── run_tests.py
    |
    └── README.md

Essential Git Commands

# Clone a repo from Azure DevOps
git clone https://dev.azure.com/MyCompany/DataPlatform/_git/data-platform

# Create a feature branch
git checkout -b feature/add-vendor-b-pipeline

# Stage and commit changes
git add .
git commit -m "Add Vendor B ingestion pipeline with schema validation"

# Push to Azure DevOps
git push origin feature/add-vendor-b-pipeline

# Create a pull request (done in Azure DevOps UI or CLI)
az repos pr create --title "Add Vendor B pipeline" --source-branch feature/add-vendor-b-pipeline

# Pull latest changes from main
git checkout main
git pull origin main

# Merge main into your branch (keep up to date)
git checkout feature/add-vendor-b-pipeline
git merge main

Branching Strategies for Data Engineering

Strategy 1: Trunk-Based (recommended for most data teams)

  main (production-ready code)
    ├── feature/add-vendor-b     (short-lived, merged within 1-2 days)
    ├── feature/fix-schema-drift (short-lived)
    └── feature/add-quality-checks (short-lived)

  Rules:
    - main is always deployable
    - Feature branches are short-lived (1-2 days max)
    - Every merge to main triggers CI/CD pipeline
    - No long-lived branches

Strategy 2: GitFlow (for teams with formal release cycles)

  main (production)
    └── develop (integration branch)
          ├── feature/add-vendor-b
          ├── feature/fix-schema-drift
          └── release/v2.1 (release candidate)

  Rules:
    - develop is the integration branch
    - Features merge into develop
    - Release branches cut from develop
    - Only release branches merge into main
    - More ceremony, better for regulated environments

Strategy 3: Environment Branches (common in ADF/Fabric)

  main (mapped to production ADF/Fabric workspace)
    ├── develop (mapped to development workspace)
    └── staging (mapped to staging workspace)

  Rules:
    - develop → staging → main (promotion flow)
    - Each branch maps to an environment
    - ADF/Fabric Git integration uses this pattern
    - Merge to staging = deploy to staging
    - Merge to main = deploy to production

Recommendation:
  Databricks + Terraform: Trunk-Based (simple, fast)
  ADF + Fabric: Environment Branches (matches Git integration)

Pull Requests and Code Reviews

Pull requests are the gate between development and production. In data engineering, they catch schema changes, SQL errors, and configuration mistakes before they reach production.

Setting up branch policies (enforce on main branch):

  1. Go to Project Settings > Repos > Policies > Branch Policies > main
  2. Enable:
     - Require minimum 1 reviewer
     - Check for linked work items (optional but good practice)
     - Check for comment resolution (all comments must be resolved)
     - Build validation: run CI pipeline on every PR
       (this catches errors BEFORE merging)

  What a good data engineering PR looks like:

  Title: "Add Vendor B daily ingestion pipeline"
  Description:
    - What: New ADF pipeline for Vendor B CSV ingestion
    - Why: Vendor B onboarding (JIRA-1234)
    - Testing: Ran in dev with 3 days of sample data
    - Impact: New table in bronze layer, no changes to existing pipelines

  Linked work item: JIRA-1234 or Azure Boards item
  Reviewers: 1-2 team members
  CI build: passing (green check)

Azure Pipelines — CI/CD Automation

Azure Pipelines is the CI/CD engine. It automates building, testing, and deploying your code whenever changes are pushed to a repository.

Pipeline types:

  Classic Pipelines:
    - Visual editor (drag-and-drop tasks)
    - Separate Build and Release pipelines
    - No version control (configuration stored in Azure DevOps, not in code)
    - Legacy -- avoid for new projects

  YAML Pipelines (recommended):
    - Pipeline defined as code (azure-pipelines.yml in your repo)
    - Version-controlled alongside your application code
    - Reviewed through pull requests (pipeline changes are code changes)
    - Supports templates, parameters, and reusable components
    - Single pipeline handles both CI and CD (multi-stage)

  Always use YAML pipelines for new projects.
  Classic pipelines exist for backward compatibility.

YAML Pipelines — The Foundation

A YAML pipeline is a file (azure-pipelines.yml) that defines what happens when code changes.

# Simple CI pipeline: runs tests on every push to main
trigger:
  branches:
    include:
      - main
  paths:
    include:
      - databricks/**    # Only trigger when Databricks files change

pool:
  vmImage: 'ubuntu-latest'    # Microsoft-hosted agent (free tier: 1800 min/month)

steps:
  - task: UsePythonVersion@0
    inputs:
      versionSpec: '3.11'
    displayName: 'Set up Python 3.11'

  - script: |
      pip install pytest databricks-sdk
      pytest tests/ -v --tb=short
    displayName: 'Run unit tests'

  - script: |
      echo "All tests passed -- ready for deployment"
    displayName: 'Summary'

Trigger Types

# Push trigger (most common)
trigger:
  branches:
    include:
      - main
      - develop
    exclude:
      - feature/*    # Don't trigger on feature branches (PRs handle that)

# Pull request trigger
pr:
  branches:
    include:
      - main
  paths:
    include:
      - terraform/**

# Scheduled trigger (nightly builds)
schedules:
  - cron: '0 2 * * *'    # 2 AM UTC daily
    displayName: 'Nightly build'
    branches:
      include:
        - main
    always: true

# Manual trigger only (no automatic triggers)
trigger: none
pr: none

Stages, Jobs, and Steps — Pipeline Anatomy

# Multi-stage pipeline: Build → Test → Deploy to Dev → Deploy to Prod
trigger:
  - main

variables:
  pythonVersion: '3.11'

stages:
  # Stage 1: Build and Test
  - stage: Build
    displayName: 'Build and Test'
    jobs:
      - job: RunTests
        pool:
          vmImage: 'ubuntu-latest'
        steps:
          - task: UsePythonVersion@0
            inputs:
              versionSpec: '$(pythonVersion)'
          - script: |
              pip install -r requirements.txt
              pytest tests/ -v
            displayName: 'Run unit tests'

  # Stage 2: Deploy to Dev
  - stage: DeployDev
    displayName: 'Deploy to Dev'
    dependsOn: Build
    condition: succeeded()
    jobs:
      - deployment: DeployToDev
        environment: 'dev'    # Azure DevOps environment (for tracking)
        pool:
          vmImage: 'ubuntu-latest'
        strategy:
          runOnce:
            deploy:
              steps:
                - script: |
                    echo "Deploying to dev environment..."
                    # Terraform apply, Databricks deploy, ADF publish, etc.
                  displayName: 'Deploy to Dev'

  # Stage 3: Deploy to Prod (requires approval)
  - stage: DeployProd
    displayName: 'Deploy to Production'
    dependsOn: DeployDev
    condition: succeeded()
    jobs:
      - deployment: DeployToProd
        environment: 'production'    # Configure approval gates on this environment
        pool:
          vmImage: 'ubuntu-latest'
        strategy:
          runOnce:
            deploy:
              steps:
                - script: |
                    echo "Deploying to production..."
                  displayName: 'Deploy to Prod'

Pipeline hierarchy:

  Pipeline (azure-pipelines.yml)
    |
    ├── Stage: Build
    |     └── Job: RunTests
    |           ├── Step: Set up Python
    |           ├── Step: Install dependencies
    |           └── Step: Run pytest
    |
    ├── Stage: DeployDev
    |     └── Deployment Job: DeployToDev
    |           ├── Step: Terraform plan
    |           └── Step: Terraform apply
    |
    └── Stage: DeployProd (requires approval)
          └── Deployment Job: DeployToProd
                ├── Step: Terraform plan
                └── Step: Terraform apply

  Stages run sequentially (or in parallel if no dependsOn)
  Jobs within a stage can run in parallel
  Steps within a job run sequentially

Service Connections — Connecting to Azure Resources

Service connections let pipelines authenticate to Azure subscriptions, Databricks workspaces, Docker registries, and other external services.

Creating a service connection (one-time setup):

  1. Project Settings > Service connections > New service connection
  2. Choose type:
     - Azure Resource Manager (most common for data engineering)
     - Databricks (for Databricks deployments)
     - Generic (for REST APIs)
  3. Authentication:
     - Automatic (creates a service principal automatically)
     - Manual (use an existing service principal)
  4. Scope:
     - Subscription level (access to entire subscription)
     - Resource group level (more secure, recommended)
  5. Name: "azure-dev-connection" or "azure-prod-connection"

  Then reference in YAML:

# Using a service connection in a pipeline
- task: AzureCLI@2
  inputs:
    azureSubscription: 'azure-prod-connection'    # Service connection name
    scriptType: 'bash'
    scriptLocation: 'inlineScript'
    inlineScript: |
      az databricks workspace show --name my-workspace --resource-group my-rg
  displayName: 'Run Azure CLI with service connection'

Variable Groups and Secrets

Variable groups store configuration values and secrets that pipelines reference. They can be linked to Azure Key Vault for secure secret management.

# Reference a variable group in your pipeline
variables:
  - group: 'dev-variables'    # Contains: STORAGE_ACCOUNT, DATABRICKS_HOST, etc.
  - name: environment
    value: 'dev'

steps:
  - script: |
      echo "Deploying to $(environment)"
      echo "Storage: $(STORAGE_ACCOUNT)"
      echo "Databricks: $(DATABRICKS_HOST)"
      # Secret variables are automatically masked in logs
    displayName: 'Deploy with variables'

Setting up a variable group:

  1. Pipelines > Library > Variable groups > New
  2. Name: "dev-variables"
  3. Add variables:
     - STORAGE_ACCOUNT = "devstorageaccount" (plain text)
     - DATABRICKS_HOST = "https://adb-123.azuredatabricks.net" (plain text)
     - DATABRICKS_TOKEN = "dapi..." (toggle lock icon = secret)
  4. Or link to Azure Key Vault:
     - Toggle "Link secrets from Azure Key Vault"
     - Select service connection and Key Vault
     - Secrets are pulled at pipeline runtime (never stored in DevOps)

  Best practice:
    Create one variable group per environment:
      dev-variables, staging-variables, prod-variables
    Link to Key Vault for all secrets
    Use plain text only for non-sensitive config (resource names, URLs)

Azure Boards — Work Tracking for Data Teams

Azure Boards provides agile work tracking with work items, sprints, backlogs, and Kanban boards.

Work item types for data engineering:

  Epic: "Onboard Vendor B Data"
    |
    ├── User Story: "Ingest Vendor B daily CSV files"
    |     ├── Task: "Create ADF pipeline for CSV ingestion"
    |     ├── Task: "Add schema validation in Databricks"
    |     └── Task: "Create bronze table and configure AutoLoader"
    |
    ├── User Story: "Transform Vendor B data to silver layer"
    |     ├── Task: "Write PySpark cleaning notebook"
    |     └── Task: "Add data quality checks"
    |
    └── Bug: "Vendor B date format changed from MM/DD to DD/MM"
          └── Task: "Update parsing logic and add format detection"

  Sprint planning for data teams:
    Sprint 1: Ingest raw data (bronze)
    Sprint 2: Transform and validate (silver)
    Sprint 3: Build aggregations (gold) + dashboards
    Sprint 4: Production hardening + monitoring

Linking work items to code:

  When you commit code or create a PR, reference the work item:
    git commit -m "Add Vendor B ingestion pipeline #1234"
    (where #1234 is the work item ID)

  This creates traceability:
    Work item 1234 → linked to commit → linked to PR → linked to build
    "Why was this pipeline changed?" → click the work item → see the full story

Azure Artifacts — Package Management

Azure Artifacts hosts private package feeds — useful for sharing Python libraries, Terraform modules, or other reusable components across teams.

Common data engineering use cases:

  1. Shared Python utility library:
     - Package: company-data-utils
     - Contains: common transformations, logging setup, config readers
     - Published to Azure Artifacts feed
     - pip install company-data-utils --index-url https://pkgs.dev.azure.com/...

  2. Shared Terraform modules:
     - Module: databricks-workspace
     - Contains: standard workspace configuration, networking, Unity Catalog setup
     - Referenced from other Terraform projects

  3. Shared SQL scripts:
     - Package: sql-quality-checks
     - Contains: reusable data quality check queries

Azure DevOps vs GitHub — Which to Choose

Recommendation for data engineering:

  Already using Azure heavily (ADF, Synapse, Databricks, Fabric)?
    → Azure DevOps (tighter integration, enterprise features)

  Using multi-cloud or open-source tools (dbt, Airflow, Spark on EMR)?
    → GitHub (broader ecosystem, GitHub Actions marketplace)

  Need Azure Boards for sprint planning?
    → Azure DevOps (GitHub Projects is less mature)

  Need GitHub Copilot for AI-assisted coding?
    → GitHub (Copilot integrates with both, but natively with GitHub)

  Many companies use both:
    GitHub for source control + GitHub Actions for CI
    Azure Boards for work tracking (linked to GitHub repos)

Setting Up Azure DevOps for a Data Engineering Project

Step-by-step setup:

  1. CREATE ORGANIZATION
     → dev.azure.com → Create Organization → "YourCompany"

  2. CREATE PROJECT
     → New Project → "DataPlatform"
     → Visibility: Private
     → Version Control: Git
     → Work Item Process: Agile

  3. CREATE REPOSITORIES
     → Repos → New Repository:
       - "terraform-infra" (Terraform code)
       - "databricks-pipelines" (notebooks, DABs)
       - "adf-pipelines" (linked via ADF Git integration)

  4. SET UP SERVICE CONNECTIONS
     → Project Settings → Service Connections:
       - "azure-dev" (Azure RM, scoped to dev resource group)
       - "azure-prod" (Azure RM, scoped to prod resource group)
       - "databricks-dev" (Databricks workspace connection)
       - "databricks-prod" (Databricks workspace connection)

  5. CREATE VARIABLE GROUPS
     → Pipelines → Library:
       - "dev-variables" (linked to dev Key Vault)
       - "prod-variables" (linked to prod Key Vault)

  6. SET UP BRANCH POLICIES
     → Project Settings → Repos → Policies → main:
       - Minimum 1 reviewer
       - Build validation (CI pipeline)
       - Comment resolution required

  7. CREATE ENVIRONMENTS
     → Pipelines → Environments:
       - "dev" (no approvals)
       - "staging" (1 approver)
       - "production" (2 approvers + business owner)

  8. CREATE PIPELINES
     → Pipelines → New Pipeline:
       - Select repo → select azure-pipelines.yml
       - CI pipeline for testing
       - CD pipeline for deployment

Common Mistakes

1. Using Classic Pipelines for new projects. Classic pipelines store configuration in Azure DevOps UI, not in code. You cannot review pipeline changes through pull requests, version them, or roll them back. Always use YAML pipelines — pipeline definitions live in your repo alongside your code.

2. Storing secrets as plain text in variable groups. Any variable not marked as secret is visible in pipeline logs and to anyone with variable group access. Always toggle the lock icon for sensitive values, or link the variable group to Azure Key Vault. Never hardcode secrets in YAML files.

3. Not setting branch policies on main. Without branch policies, anyone can push directly to main without review or testing. One bad push can break production. Enable minimum reviewers, build validation (CI runs on every PR), and comment resolution on the main branch.

4. Creating one giant repository for everything. A single repo with ADF pipelines, Databricks notebooks, Terraform configs, Python utilities, and Power BI models becomes unmanageable. Split by concern: one repo for infrastructure (Terraform), one for data pipelines (Databricks/ADF), one for analytics (dbt/Power BI).

5. Using organization-scoped service connections. A service connection scoped to the entire Azure subscription gives every pipeline access to every resource. Scope connections to specific resource groups (dev-rg, prod-rg) to limit blast radius if a pipeline is misconfigured.

6. Not using environments with approval gates. Without approval gates on the production environment, a CI/CD pipeline can deploy to production automatically without human verification. Configure the production environment to require at least one approver, especially for infrastructure changes.

7. Hardcoding environment-specific values in pipeline YAML. Putting STORAGE_ACCOUNT = "prodstorageaccount" directly in YAML means changing environments requires editing code. Use variable groups per environment (dev-variables, staging-variables, prod-variables) and reference them with $(VARIABLE_NAME).

8. Not linking work items to commits and PRs. Without traceability, you cannot answer “why was this pipeline changed?” or “which sprint delivered this feature.” Reference work item IDs in commit messages (#1234) and link PRs to work items. Azure DevOps automatically creates bidirectional links.

Interview Questions

Q: What is Azure DevOps and what are its five core services? A: Azure DevOps is Microsoft’s integrated platform for software development lifecycle management. Its five services are Azure Repos (Git-based source control), Azure Pipelines (CI/CD automation), Azure Boards (agile work tracking with sprints and Kanban boards), Azure Artifacts (package management for private feeds), and Azure Test Plans (manual and automated testing). For data engineering, Repos and Pipelines are the most critical — they version-control pipeline code and automate deployments to Azure services like Data Factory, Databricks, and Fabric.

Q: What is the difference between Classic Pipelines and YAML Pipelines? A: Classic Pipelines use a visual drag-and-drop editor in Azure DevOps UI. Their configuration is stored in Azure DevOps, not in source control. YAML Pipelines define the pipeline as code in an azure-pipelines.yml file that lives in your repository. YAML pipelines are version-controlled, reviewable through pull requests, and can use templates for reusability. Always use YAML pipelines for new projects because pipeline changes go through the same review process as code changes.

Q: What is a service connection and why is it important? A: A service connection stores authentication credentials that allow Azure Pipelines to access external services like Azure subscriptions, Databricks workspaces, or Docker registries. Instead of hardcoding credentials in pipeline YAML, you reference the service connection by name. This centralizes credential management, limits access scope (you can restrict a connection to a specific resource group), and prevents secrets from appearing in pipeline logs. Create separate connections per environment (dev, staging, prod) with appropriate scope.

Q: What are variable groups and how do they relate to Key Vault? A: Variable groups store configuration values and secrets that pipelines reference using $(VARIABLE_NAME) syntax. You can define variables directly in the group (with the option to mark them as secrets) or link the group to an Azure Key Vault, which pulls secrets at pipeline runtime. Key Vault linking is the most secure approach because secrets are never stored in Azure DevOps — they are fetched on demand and automatically masked in pipeline logs. Create separate variable groups per environment (dev-variables, staging-variables, prod-variables).

Q: What are branch policies and why should you enforce them? A: Branch policies are rules that must be satisfied before code can be merged into a protected branch (usually main). Common policies include minimum number of reviewers, build validation (CI pipeline must pass), comment resolution (all review comments must be resolved), and linked work items. For data engineering, build validation is critical — it runs unit tests, linting, and validation on every pull request, catching errors before they reach production.

Q: What branching strategy would you recommend for a data engineering team? A: It depends on the tools. For Databricks and Terraform projects, trunk-based development works best: short-lived feature branches merged into main within 1-2 days, with CI/CD triggered on every merge. For ADF and Fabric projects, environment branches (develop → staging → main) align with their Git integration model, where each branch maps to a workspace. The key principle is that main should always represent production-ready code, and every change goes through a pull request with CI validation.

Q: How would you set up Azure DevOps for a new data engineering project? A: Create an organization and project. Set up separate repositories by concern (terraform-infra, databricks-pipelines, adf-pipelines). Create service connections scoped to specific resource groups per environment. Set up variable groups linked to Key Vault for secrets. Enable branch policies on main (minimum reviewers, build validation). Create environments (dev, staging, production) with approval gates on production. Finally, create YAML pipelines for CI (test on every PR) and CD (deploy on merge to main). This gives you version-controlled code, automated testing, environment promotion, and audit trails from day one.

Wrapping Up

Azure DevOps provides the complete platform for data engineering teams to version-control code, automate deployments, track work, and manage packages. Repos stores your Terraform configs, Databricks notebooks, and pipeline definitions. Pipelines automates the build-test-deploy cycle. Boards tracks your sprints. And service connections and variable groups keep your credentials secure.

In the next post, we will dive deep into YAML pipelines — triggers, templates, parameters, conditional logic, multi-stage deployments, and the patterns that production data engineering teams use daily.

Related posts: – CI/CD for ADF with Azure DevOps – CI/CD for ADF with GitHub – Databricks Git Integration & CI/CD – Databricks Asset Bundles (DABs) – Fabric Git Integration & Deployment Pipelines