Terraform for Data Engineers: HCL, Providers, Resources, Variables, State Management, Provisioning Storage Accounts, Databricks Workspaces, Unity Catalog, Key Vaults, and Multi-Environment Deployment

Table of Contents

• What Is Terraform and Why Data Engineers Need It
• Installing Terraform and Getting Started
• HCL Fundamentals — The Language of Infrastructure
• Providers — Connecting Terraform to Cloud Platforms
• Resources — Creating Infrastructure
• Data Sources — Reading Existing Infrastructure
• Variables — Making Configuration Flexible
• Outputs — Exposing Values
• Locals — Computed Values
• The Terraform Workflow — Init, Plan, Apply, Destroy
• State Management — The Brain of Terraform
• Provisioning Azure Resources for Data Engineering
• Provisioning a Databricks Workspace
• Provisioning Databricks Resources — Clusters, Jobs, Unity Catalog
• Multi-Environment Deployment with tfvars
• Complete Data Platform Project Structure
• Common Mistakes
• Interview Questions
• Wrapping Up

In the previous post, we covered YAML pipelines for automating deployments. But what are we deploying? Most data engineering teams deploy infrastructure — storage accounts, Databricks workspaces, Key Vaults, networking, Unity Catalog configurations, and clusters. Terraform lets you define all of this as code, version-control it in Git, review it through pull requests, and deploy it through YAML pipelines.

Analogy — A blueprint for a building. Clicking through the Azure portal to create resources is like building a house by pointing and telling workers what to do (“put a wall here, add a door there”). It works for one house, but what if you need three identical houses (dev, staging, prod)? Or if you need to rebuild after a disaster? Terraform is the architect’s blueprint — a precise, repeatable document that describes exactly what the building looks like. Run terraform apply and the construction crew (Azure, AWS, GCP) builds it exactly as drawn. Run it again, and nothing changes (idempotent). Change the blueprint (edit the code), run terraform apply again, and only the changed parts are updated. Tear it all down with terraform destroy.

What Is Terraform and Why Data Engineers Need It

Terraform is an infrastructure as code (IaC) tool by HashiCorp that lets you define cloud infrastructure in declarative configuration files. Instead of clicking through the Azure portal, you write code that describes what you want, and Terraform creates it.

Why data engineers need Terraform:

  Without Terraform (ClickOps):
    - Create storage account manually in Azure portal
    - Create Databricks workspace manually
    - Create Key Vault manually
    - Repeat for dev, staging, prod (3x the clicking)
    - Document every step (or forget and lose the configuration)
    - No audit trail of who changed what
    - Recreating the environment from scratch takes days

  With Terraform:
    - Write once: storage account, Databricks, Key Vault in code
    - Deploy to dev: terraform apply -var-file=dev.tfvars
    - Deploy to staging: terraform apply -var-file=staging.tfvars
    - Deploy to prod: terraform apply -var-file=prod.tfvars
    - Version-controlled in Git (full audit trail)
    - Review infrastructure changes through pull requests
    - Recreate the entire environment in minutes
    - Consistent: dev, staging, prod are identical by design

  Key Terraform concepts:
    - Declarative: you describe WHAT you want, not HOW to create it
    - Idempotent: running apply twice produces the same result
    - State-aware: Terraform tracks what exists and only changes what is different
    - Multi-cloud: same language for Azure, AWS, GCP, Databricks, Snowflake

Installing Terraform and Getting Started

# macOS (Homebrew)
brew tap hashicorp/tap
brew install hashicorp/tap/terraform
terraform --version
# Terraform v1.9.x

# Windows (Chocolatey)
choco install terraform

# Linux (Ubuntu/Debian)
wget -O - https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform

# Verify installation
terraform --version

HCL Fundamentals — The Language of Infrastructure

Terraform uses HCL (HashiCorp Configuration Language) — a declarative language designed for infrastructure definition. It looks like JSON but is more readable.

# This is an HCL comment

# Block syntax: type "label1" "label2" { ... }
resource "azurerm_resource_group" "main" {
  name     = "rg-data-platform-dev"
  location = "Canada Central"

  tags = {
    environment = "dev"
    team        = "data-engineering"
    managed_by  = "terraform"
  }
}

# HCL data types:
#   String:  "hello"
#   Number:  42
#   Boolean: true / false
#   List:    ["a", "b", "c"]
#   Map:     { key1 = "val1", key2 = "val2" }
#   null:    null (resource default)

Standard file layout:

  main.tf         -- Primary resources (the main infrastructure)
  variables.tf    -- Input variable declarations
  outputs.tf      -- Output value declarations
  providers.tf    -- Provider configuration (Azure, Databricks)
  versions.tf     -- Terraform and provider version constraints
  terraform.tfvars -- Variable values (or dev.tfvars, prod.tfvars)
  locals.tf       -- Computed local values

  All .tf files in a directory are loaded together.
  File names are conventions -- Terraform does not care what you name them.
  But following these conventions makes code readable for any Terraform user.

Providers — Connecting Terraform to Cloud Platforms

A provider is a plugin that lets Terraform manage resources on a specific platform. For data engineering, you typically use 2-4 providers:

# versions.tf -- Pin Terraform and provider versions
terraform {
  required_version = ">= 1.7.0"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 4.0"       # Latest 4.x (Azure)
    }
    databricks = {
      source  = "databricks/databricks"
      version = "~> 1.50"      # Latest 1.50.x (Databricks)
    }
    azuread = {
      source  = "hashicorp/azuread"
      version = "~> 2.50"      # Azure Active Directory
    }
  }
}

# providers.tf -- Configure each provider
provider "azurerm" {
  features {}
  subscription_id = var.subscription_id
  # Authentication: uses Azure CLI (az login) locally,
  # or service principal in CI/CD pipelines
}

provider "databricks" {
  host = azurerm_databricks_workspace.main.workspace_url
  # Authentication: uses Azure CLI or service principal
}

provider "azuread" {
  # Uses Azure CLI authentication
}

Common providers for data engineering:

  azurerm    -- Azure resources (storage, networking, Key Vault, ADF)
  databricks -- Databricks resources (clusters, jobs, Unity Catalog)
  azuread    -- Azure AD (service principals, groups)
  snowflake  -- Snowflake resources (databases, schemas, warehouses)
  aws        -- AWS resources (S3, Glue, IAM)
  random     -- Random values (passwords, suffixes for unique names)
  null       -- Placeholder resources for scripting

Resources — Creating Infrastructure

Resources are the core building blocks. Each resource block creates one piece of infrastructure.

# Syntax: resource "provider_type" "local_name" { ... }

# Create a resource group
resource "azurerm_resource_group" "main" {
  name     = "rg-data-platform-${var.environment}"
  location = var.location

  tags = local.common_tags
}

# Create a storage account (Data Lake)
resource "azurerm_storage_account" "datalake" {
  name                     = "stdatalake${var.environment}001"
  resource_group_name      = azurerm_resource_group.main.name  # Reference another resource
  location                 = azurerm_resource_group.main.location
  account_tier             = "Standard"
  account_replication_type = "LRS"
  account_kind             = "StorageV2"
  is_hns_enabled           = true    # Hierarchical namespace = ADLS Gen2

  tags = local.common_tags
}

# Create containers (bronze, silver, gold)
resource "azurerm_storage_container" "bronze" {
  name                  = "bronze"
  storage_account_id    = azurerm_storage_account.datalake.id
  container_access_type = "private"
}

resource "azurerm_storage_container" "silver" {
  name                  = "silver"
  storage_account_id    = azurerm_storage_account.datalake.id
  container_access_type = "private"
}

resource "azurerm_storage_container" "gold" {
  name                  = "gold"
  storage_account_id    = azurerm_storage_account.datalake.id
  container_access_type = "private"
}

# Create a Key Vault
resource "azurerm_key_vault" "main" {
  name                = "kv-dataplatform-${var.environment}"
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location
  tenant_id           = data.azurerm_client_config.current.tenant_id
  sku_name            = "standard"

  purge_protection_enabled = true

  tags = local.common_tags
}

Resource references:

  Resources reference each other using: resource_type.local_name.attribute

  azurerm_resource_group.main.name → "rg-data-platform-dev"
  azurerm_storage_account.datalake.id → "/subscriptions/.../storageAccounts/..."
  azurerm_key_vault.main.vault_uri → "https://kv-dataplatform-dev.vault.azure.net/"

  Terraform uses these references to:
  1. Build a dependency graph (knows to create resource group before storage account)
  2. Pass values between resources (storage account uses resource group's name)
  3. Detect changes (if resource group name changes, storage account must be updated)

Data Sources — Reading Existing Infrastructure

Data sources read information about resources that already exist, without creating or modifying them.

# Read the current Azure client config (who am I?)
data "azurerm_client_config" "current" {}

# Read an existing resource group (created outside Terraform)
data "azurerm_resource_group" "existing" {
  name = "rg-shared-services"
}

# Read an existing Key Vault secret
data "azurerm_key_vault_secret" "db_password" {
  name         = "database-password"
  key_vault_id = azurerm_key_vault.main.id
}

# Use data source values
resource "azurerm_key_vault_access_policy" "terraform" {
  key_vault_id = azurerm_key_vault.main.id
  tenant_id    = data.azurerm_client_config.current.tenant_id
  object_id    = data.azurerm_client_config.current.object_id

  secret_permissions = ["Get", "List", "Set", "Delete"]
}

Variables — Making Configuration Flexible

Variables make your Terraform code reusable across environments.

# variables.tf -- Declare variables with types and defaults
variable "environment" {
  description = "Environment name (dev, staging, prod)"
  type        = string
  default     = "dev"

  validation {
    condition     = contains(["dev", "staging", "prod"], var.environment)
    error_message = "Environment must be dev, staging, or prod."
  }
}

variable "location" {
  description = "Azure region for all resources"
  type        = string
  default     = "Canada Central"
}

variable "subscription_id" {
  description = "Azure subscription ID"
  type        = string
  sensitive   = true    # Masked in terraform plan output
}

variable "storage_containers" {
  description = "List of storage containers to create"
  type        = list(string)
  default     = ["bronze", "silver", "gold"]
}

variable "databricks_sku" {
  description = "Databricks workspace SKU"
  type        = string
  default     = "premium"
}

variable "tags" {
  description = "Common tags for all resources"
  type        = map(string)
  default     = {}
}

# dev.tfvars -- Values for dev environment
environment     = "dev"
location        = "Canada Central"
subscription_id = "12345678-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
databricks_sku  = "premium"

tags = {
  environment = "dev"
  team        = "data-engineering"
  cost_center = "DE-001"
}

# prod.tfvars -- Values for production
environment     = "prod"
location        = "Canada Central"
subscription_id = "87654321-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
databricks_sku  = "premium"

tags = {
  environment = "prod"
  team        = "data-engineering"
  cost_center = "DE-001"
}

Ways to set variable values (in priority order):

  1. Command line:       terraform apply -var="environment=prod"
  2. .tfvars file:       terraform apply -var-file="prod.tfvars"
  3. Environment var:    export TF_VAR_environment="prod"
  4. Default value:      default = "dev" in variables.tf
  5. Interactive prompt: Terraform asks if no value is set

  For CI/CD pipelines, use .tfvars files per environment:
    terraform apply -var-file="environments/dev.tfvars"
    terraform apply -var-file="environments/prod.tfvars"

Outputs — Exposing Values

Outputs expose resource attributes for use by other Terraform configurations, scripts, or pipelines.

# outputs.tf
output "resource_group_name" {
  description = "Name of the resource group"
  value       = azurerm_resource_group.main.name
}

output "storage_account_name" {
  description = "Name of the data lake storage account"
  value       = azurerm_storage_account.datalake.name
}

output "databricks_workspace_url" {
  description = "URL of the Databricks workspace"
  value       = azurerm_databricks_workspace.main.workspace_url
}

output "key_vault_uri" {
  description = "URI of the Key Vault"
  value       = azurerm_key_vault.main.vault_uri
}

# After terraform apply, access outputs:
# terraform output databricks_workspace_url
# → "https://adb-1234567890.10.azuredatabricks.net"

Locals — Computed Values

Locals define computed values that simplify your configuration.

# locals.tf
locals {
  # Naming convention
  name_prefix = "dp-${var.environment}"

  # Common tags applied to all resources
  common_tags = merge(var.tags, {
    managed_by  = "terraform"
    environment = var.environment
    project     = "data-platform"
  })

  # Computed resource names
  resource_group_name  = "rg-${local.name_prefix}"
  storage_account_name = "st${replace(local.name_prefix, "-", "")}001"
  key_vault_name       = "kv-${local.name_prefix}"
  databricks_name      = "dbw-${local.name_prefix}"
}

# Use locals in resources
resource "azurerm_resource_group" "main" {
  name     = local.resource_group_name
  location = var.location
  tags     = local.common_tags
}

The Terraform Workflow — Init, Plan, Apply, Destroy

# Step 1: Initialize (download providers, set up backend)
terraform init
# Downloads azurerm, databricks providers
# Configures remote state backend (if configured)
# Only needed once per project (or when providers change)

# Step 2: Format (standardize code style)
terraform fmt
# Reformats .tf files to canonical style
# Run before every commit

# Step 3: Validate (check syntax)
terraform validate
# Checks HCL syntax and resource configuration
# Does NOT check against Azure -- just local validation

# Step 4: Plan (preview changes)
terraform plan -var-file="dev.tfvars"
# Shows what Terraform WILL do:
#   + create (new resources)
#   ~ update (changed resources)
#   - destroy (removed resources)
# NEVER makes actual changes

# Step 5: Apply (execute changes)
terraform apply -var-file="dev.tfvars"
# Creates/updates/destroys resources to match configuration
# Asks for confirmation (or use -auto-approve in CI/CD)

# Step 6: Destroy (tear down everything)
terraform destroy -var-file="dev.tfvars"
# Destroys ALL resources managed by this configuration
# Use carefully -- this deletes everything
# Useful for: tearing down dev/test environments after testing

The plan → apply workflow:

  terraform plan -var-file="dev.tfvars" -out=plan.tfplan
  # Saves the plan to a file

  terraform apply plan.tfplan
  # Applies EXACTLY what was planned (no surprises)
  # This is the safest approach for production deployments

  In CI/CD:
    Stage 1 (Plan):  terraform plan -out=plan.tfplan → publish artifact
    Stage 2 (Apply): download artifact → terraform apply plan.tfplan
    This ensures the reviewed plan is the exact plan that gets applied

State Management — The Brain of Terraform

Terraform state is a JSON file that tracks which real-world resources correspond to your configuration. Without state, Terraform does not know what exists.

Analogy — An inventory ledger. The state file is like a warehouse inventory ledger. It records “Row 3, Shelf A: Widget X” (resource group: rg-data-platform-dev). When you run terraform plan, Terraform compares the ledger (state) with the actual warehouse (Azure) and the desired layout (your .tf files). If something is missing, it adds it. If something changed, it updates it. If something is in the ledger but not in your configuration, it removes it.

# Remote state backend (ALWAYS use remote state for team projects)
# Put this in versions.tf or backend.tf
terraform {
  backend "azurerm" {
    resource_group_name  = "rg-terraform-state"
    storage_account_name = "stterraformstate001"
    container_name       = "tfstate"
    key                  = "dataplatform.tfstate"
  }
}

# Initialize with backend config (useful for multiple environments)
terraform init -backend-config="environments/dev/backend.conf"

# environments/dev/backend.conf
resource_group_name  = "rg-terraform-state"
storage_account_name = "stterraformstate001"
container_name       = "tfstate"
key                  = "dataplatform-dev.tfstate"

# environments/prod/backend.conf
resource_group_name  = "rg-terraform-state"
storage_account_name = "stterraformstate001"
container_name       = "tfstate"
key                  = "dataplatform-prod.tfstate"

State management rules:

  1. NEVER use local state for team projects
     Local state = terraform.tfstate on your laptop
     If your laptop crashes, state is lost, Terraform can't manage resources

  2. ALWAYS use remote state (Azure Blob, S3, Terraform Cloud)
     Remote state is shared, locked, and versioned
     Multiple team members can work on the same infrastructure

  3. NEVER edit state manually
     Use terraform state commands if you need to move/remove resources
     Manual edits corrupt state and cause unpredictable behavior

  4. Enable state locking
     Azure Blob Storage uses lease locks automatically
     Prevents two people from applying at the same time

  5. Enable state versioning
     Azure Storage versioning lets you roll back to a previous state
     Essential for recovering from accidental state corruption

Provisioning Azure Resources for Data Engineering

# Complete data platform infrastructure
# main.tf

# Resource Group
resource "azurerm_resource_group" "main" {
  name     = local.resource_group_name
  location = var.location
  tags     = local.common_tags
}

# ADLS Gen2 Storage Account (Data Lake)
resource "azurerm_storage_account" "datalake" {
  name                     = local.storage_account_name
  resource_group_name      = azurerm_resource_group.main.name
  location                 = azurerm_resource_group.main.location
  account_tier             = "Standard"
  account_replication_type = var.environment == "prod" ? "GRS" : "LRS"
  account_kind             = "StorageV2"
  is_hns_enabled           = true    # ADLS Gen2

  tags = local.common_tags
}

# Create medallion containers
resource "azurerm_storage_container" "layers" {
  for_each              = toset(["bronze", "silver", "gold", "landing"])
  name                  = each.value
  storage_account_id    = azurerm_storage_account.datalake.id
  container_access_type = "private"
}

# Key Vault
resource "azurerm_key_vault" "main" {
  name                     = local.key_vault_name
  resource_group_name      = azurerm_resource_group.main.name
  location                 = azurerm_resource_group.main.location
  tenant_id                = data.azurerm_client_config.current.tenant_id
  sku_name                 = "standard"
  purge_protection_enabled = var.environment == "prod"

  tags = local.common_tags
}

# Key Vault access policy for Terraform service principal
resource "azurerm_key_vault_access_policy" "terraform" {
  key_vault_id = azurerm_key_vault.main.id
  tenant_id    = data.azurerm_client_config.current.tenant_id
  object_id    = data.azurerm_client_config.current.object_id

  secret_permissions = ["Get", "List", "Set", "Delete", "Purge"]
  key_permissions    = ["Get", "List", "Create"]
}

# Store a secret in Key Vault
resource "azurerm_key_vault_secret" "storage_key" {
  name         = "storage-account-key"
  value        = azurerm_storage_account.datalake.primary_access_key
  key_vault_id = azurerm_key_vault.main.id

  depends_on = [azurerm_key_vault_access_policy.terraform]
}

Provisioning a Databricks Workspace

# Databricks workspace
resource "azurerm_databricks_workspace" "main" {
  name                = local.databricks_name
  resource_group_name = azurerm_resource_group.main.name
  location            = azurerm_resource_group.main.location
  sku                 = var.databricks_sku    # "premium" for Unity Catalog

  tags = local.common_tags
}

# Output the workspace URL
output "databricks_workspace_url" {
  value = "https://${azurerm_databricks_workspace.main.workspace_url}"
}

output "databricks_workspace_id" {
  value = azurerm_databricks_workspace.main.workspace_id
}

Provisioning Databricks Resources — Clusters, Jobs, Unity Catalog

After the workspace is created, use the Databricks provider to manage resources inside it.

# Configure Databricks provider (uses workspace created above)
provider "databricks" {
  host                        = azurerm_databricks_workspace.main.workspace_url
  azure_workspace_resource_id = azurerm_databricks_workspace.main.id
}

# Create a cluster
resource "databricks_cluster" "shared" {
  cluster_name            = "shared-${var.environment}"
  spark_version           = "15.4.x-scala2.12"
  node_type_id            = "Standard_DS3_v2"
  autotermination_minutes = 20
  num_workers             = 0                    # Single node for dev

  spark_conf = {
    "spark.databricks.cluster.profile" = "singleNode"
    "spark.master"                     = "local[*]"
  }

  custom_tags = {
    "ResourceClass" = "SingleNode"
    "Environment"   = var.environment
  }
}

# Create a SQL Warehouse (serverless)
resource "databricks_sql_endpoint" "main" {
  name             = "sql-warehouse-${var.environment}"
  cluster_size     = "2X-Small"
  max_num_clusters = 1
  auto_stop_mins   = 10

  tags {
    custom_tags {
      key   = "Environment"
      value = var.environment
    }
  }
}

# Create a Unity Catalog storage credential
resource "databricks_storage_credential" "main" {
  name = "sc-${var.environment}"

  azure_managed_identity {
    access_connector_id = azurerm_databricks_access_connector.main.id
  }
}

# Create an external location
resource "databricks_external_location" "datalake" {
  name            = "el-datalake-${var.environment}"
  url             = "abfss://bronze@${azurerm_storage_account.datalake.name}.dfs.core.windows.net/"
  credential_name = databricks_storage_credential.main.name
}

# Create a catalog
resource "databricks_catalog" "main" {
  name    = "${var.environment}_catalog"
  comment = "Data catalog for ${var.environment}"
}

# Create schemas
resource "databricks_schema" "layers" {
  for_each     = toset(["bronze", "silver", "gold"])
  catalog_name = databricks_catalog.main.name
  name         = each.value
  comment      = "${each.value} layer for ${var.environment}"
}

Multi-Environment Deployment with tfvars

Project structure for multi-environment:

  terraform/
    ├── main.tf              # Resources
    ├── variables.tf         # Variable declarations
    ├── outputs.tf           # Outputs
    ├── providers.tf         # Provider config
    ├── versions.tf          # Version constraints + backend
    ├── locals.tf            # Computed values
    |
    ├── environments/
    |     ├── dev/
    |     |     ├── dev.tfvars        # Dev variable values
    |     |     └── backend.conf      # Dev state backend config
    |     ├── staging/
    |     |     ├── staging.tfvars
    |     |     └── backend.conf
    |     └── prod/
    |           ├── prod.tfvars
    |           └── backend.conf
    |
    └── modules/             # Reusable modules (covered in next post)
          ├── storage/
          ├── databricks/
          └── keyvault/

  Deploy to dev:
    terraform init -backend-config="environments/dev/backend.conf"
    terraform plan -var-file="environments/dev/dev.tfvars"
    terraform apply -var-file="environments/dev/dev.tfvars"

  Deploy to prod:
    terraform init -backend-config="environments/prod/backend.conf"
    terraform plan -var-file="environments/prod/prod.tfvars"
    terraform apply -var-file="environments/prod/prod.tfvars"

Complete Data Platform Project Structure

# Full main.tf for a data engineering platform

data "azurerm_client_config" "current" {}

# 1. Resource Group
resource "azurerm_resource_group" "main" {
  name     = "rg-${local.name_prefix}"
  location = var.location
  tags     = local.common_tags
}

# 2. ADLS Gen2 (Data Lake)
resource "azurerm_storage_account" "datalake" {
  name                     = "st${replace(local.name_prefix, "-", "")}lake"
  resource_group_name      = azurerm_resource_group.main.name
  location                 = var.location
  account_tier             = "Standard"
  account_replication_type = var.environment == "prod" ? "GRS" : "LRS"
  account_kind             = "StorageV2"
  is_hns_enabled           = true
  tags                     = local.common_tags
}

# 3. Containers
resource "azurerm_storage_container" "layers" {
  for_each              = toset(var.storage_containers)
  name                  = each.value
  storage_account_id    = azurerm_storage_account.datalake.id
  container_access_type = "private"
}

# 4. Key Vault
resource "azurerm_key_vault" "main" {
  name                     = "kv-${local.name_prefix}"
  resource_group_name      = azurerm_resource_group.main.name
  location                 = var.location
  tenant_id                = data.azurerm_client_config.current.tenant_id
  sku_name                 = "standard"
  purge_protection_enabled = var.environment == "prod"
  tags                     = local.common_tags
}

# 5. Databricks Access Connector (for Unity Catalog managed identity)
resource "azurerm_databricks_access_connector" "main" {
  name                = "ac-${local.name_prefix}"
  resource_group_name = azurerm_resource_group.main.name
  location            = var.location

  identity {
    type = "SystemAssigned"
  }
  tags = local.common_tags
}

# 6. Grant Access Connector access to storage
resource "azurerm_role_assignment" "connector_to_storage" {
  scope                = azurerm_storage_account.datalake.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_databricks_access_connector.main.identity[0].principal_id
}

# 7. Databricks Workspace
resource "azurerm_databricks_workspace" "main" {
  name                = "dbw-${local.name_prefix}"
  resource_group_name = azurerm_resource_group.main.name
  location            = var.location
  sku                 = var.databricks_sku
  tags                = local.common_tags
}

# 8. Terraform State Backend Storage (create ONCE, manually or in a separate project)
# resource "azurerm_storage_account" "tfstate" { ... }

Common Mistakes

1. Using local state for team projects. Local state lives on your laptop. If you lose the laptop, Terraform cannot manage the infrastructure. Always use remote state (Azure Blob Storage with locking) for any project with more than one contributor. Create the state storage account manually before running terraform init.

2. Not pinning provider versions. Writing version = ">= 4.0" means any 4.x version is accepted. A provider update could introduce breaking changes. Pin to minor versions: version = "~> 4.0" (allows 4.0.x but not 4.1.0) to balance stability and security patches.

3. Hardcoding values instead of using variables. Writing name = "rg-data-platform-dev" makes the code unusable for staging and prod. Use variables (var.environment) and locals (local.name_prefix) to make the same code deploy to any environment.

4. Running terraform apply without reviewing the plan. Always run terraform plan first and review the output. Look for resources being destroyed unexpectedly (marked with -). In CI/CD, save the plan to a file (-out=plan.tfplan) and apply that exact plan to prevent drift between plan and apply.

5. Not using for_each for similar resources. Creating three separate azurerm_storage_container blocks for bronze, silver, and gold is repetitive. Use for_each = toset(["bronze", "silver", "gold"]) to create them from a list. Adding a new container is a one-line change.

6. Storing secrets in .tfvars files committed to Git. Variable files are often committed to Git for convenience, but secrets (passwords, keys, tokens) should never be in Git. Use Azure Key Vault data sources, environment variables (TF_VAR_secret), or Azure Pipeline variable groups to pass secrets at runtime.

7. Not using remote state locking. Without locking, two team members running terraform apply simultaneously can corrupt state. Azure Blob Storage backend uses blob lease locking automatically, but you must ensure the storage account is configured correctly.

8. Destroying production resources accidentally. A terraform destroy or removing a resource block from .tf files destroys real infrastructure. Use lifecycle { prevent_destroy = true } on critical production resources to prevent accidental deletion. Always review plan output for - (destroy) markers before applying.

Interview Questions

Q: What is Terraform and how does it differ from ARM templates or Azure CLI? A: Terraform is a multi-cloud infrastructure as code tool using HCL (HashiCorp Configuration Language). ARM templates are Azure-only JSON files. Azure CLI runs imperative commands. Terraform is declarative (you describe the desired state, Terraform figures out the steps), idempotent (running apply twice produces the same result), and multi-cloud (same language for Azure, AWS, GCP, Databricks). Terraform also tracks state, enabling it to detect drift and make incremental changes, unlike CLI scripts that must handle idempotency manually.

Q: What is Terraform state and why is remote state important? A: Terraform state is a JSON file that maps your configuration to real-world resources. It records resource IDs, attributes, and dependencies. Without state, Terraform cannot know what exists and would try to create everything from scratch. Remote state stores this file in a shared location (Azure Blob Storage, S3, Terraform Cloud) with locking and versioning. This enables team collaboration, prevents concurrent modifications, and protects against laptop loss. Local state should only be used for individual experimentation.

Q: Explain the difference between variables, locals, and outputs in Terraform. A: Variables are input parameters that make configurations flexible — they are set by the user via .tfvars files, command-line flags, or environment variables. Locals are computed values derived from variables or other expressions — they simplify complex expressions and enforce naming conventions. Outputs expose resource attributes after apply — they can be read by other Terraform configurations, scripts, or CI/CD pipelines. Variables go in, locals compute, outputs come out.

Q: How do you manage multiple environments (dev, staging, prod) with Terraform? A: Use a single set of .tf files with environment-specific .tfvars files (dev.tfvars, prod.tfvars) that provide different values for variables like names, SKUs, and regions. Use separate state files per environment via backend configuration (backend.conf per environment). Deploy with terraform apply -var-file="environments/dev/dev.tfvars". This ensures all environments use the same infrastructure code but with environment-appropriate values. The code is written once and parameterized for reuse.

Q: What is the for_each meta-argument and when would you use it? A: for_each creates multiple instances of a resource from a set or map. Instead of writing three separate storage container blocks, you write one block with for_each = toset(["bronze", "silver", "gold"]) and each container is created from the list. Adding a new container requires only adding a string to the list. Each instance is independently managed in state, so removing one does not affect others. Use for_each whenever you have multiple similar resources that differ by name or a few attributes.

Q: How would you provision a complete data engineering platform with Terraform? A: Create a resource group, then an ADLS Gen2 storage account with hierarchical namespace enabled, storage containers for medallion layers (bronze, silver, gold), an Azure Key Vault for secrets, a Databricks Access Connector with system-assigned managed identity, RBAC role assignments granting the connector access to storage, and a Databricks workspace with premium SKU. Use the Databricks provider to configure Unity Catalog storage credentials, external locations, catalogs, and schemas inside the workspace. Parameterize everything with variables and deploy with environment-specific .tfvars files.

Q: What does terraform plan show and why should you save it to a file? A: terraform plan shows what Terraform will do without making changes: resources to create (+), update (~), or destroy (-). Saving the plan to a file with -out=plan.tfplan captures the exact set of changes. When you run terraform apply plan.tfplan, Terraform applies that exact plan, preventing any drift between when you reviewed the plan and when you applied it. This is critical in CI/CD where the plan stage and apply stage might run minutes or hours apart with potential concurrent changes.

Wrapping Up

Terraform transforms data platform infrastructure from manual portal clicks into version-controlled, reviewable, repeatable code. Providers connect to Azure, Databricks, and Snowflake. Resources define what to create. Variables make it flexible. State tracks what exists. And the plan-apply workflow ensures you always know what will change before it happens.

In the next post, we will cover Terraform modules — writing reusable infrastructure components and deploying them through the YAML pipelines we built in the previous post, creating a complete infrastructure CI/CD pipeline.

Related posts: – Azure DevOps Overview – YAML Pipelines Deep Dive – Azure Key Vault – ADLS Gen2 Guide – Databricks Unity Catalog