☁️
AZURE
Azure for Data Engineers
Data Factory, ADLS Gen2, Synapse, Key Vault, RBAC, file formats, pipelines, and everything Azure for data engineering.
37 in-depth tutorials
Cloud computing fundamentals: on-premises vs cloud, IaaS vs PaaS vs SaaS, and the shared responsibility model.
Everything a data engineer needs to know before starting an Azure project. Covers subscriptions, resource groups, IAM roles, Managed Identity, Key Vault, regions, tags, cost management, networking basics, Azure CLI…
Azure Data Factory from scratch: pipelines, activities, linked services, and your first data pipeline.
Complete guide to setting up Azure Synapse Analytics. Covers workspace creation, Synapse Studio walkthrough, Serverless SQL Pool, Dedicated SQL Pool, Spark Pool, building your first pipeline, Git integration, Managed…
A practical comparison of ADF and Synapse Pipelines. Same engine, different context. UI differences, features, pricing, and when to use each.
Master the differences between databases and data warehouses, OLTP vs OLAP, normalized vs star schema, row vs columnar storage.
Everything about Azure Blob Storage – hierarchy, blob types, access tiers, authentication, lifecycle management, and ADLS Gen2 comparison.
ADLS Gen2 architecture: hierarchical namespaces, access tiers, and the foundation for data lakes.
Complete guide to Azure SQL Database for data engineers. Covers service tiers, DTU vs vCore, authentication methods, Managed Identity, firewall rules, stored procedures, temporal tables, pipeline integration, and cost…
Master Azure networking for data engineering. VNets, Subnets, NSGs (inbound/outbound rules), Private Endpoints, Service Endpoints, VNet Peering, VPN Gateway, ExpressRoute, DNS, and production network architecture.
File formats compared: CSV, Parquet, Delta, Avro, ORC, and JSON with decision framework.
Master Azure RBAC with the hotel key card analogy. Every role organized by service: Storage (the confusing ones explained), SQL, Synapse, Databricks, Data Factory, Key Vault, Networking, and Compute.
13Azure Connections and Authentication
The Azure connections reference card for data engineers. Five authentication methods explained with building key analogies (master key, visitor badge, facial recognition, employee badge, full address).
Learn how to build a metadata-driven pipeline in Azure Data Factory using the Lookup, ForEach, and Copy pattern.
Build a metadata-driven pipeline in Azure Synapse Analytics with Parquet output and stored procedure audit logging. Track rows copied, duration, and errors for every table. Step-by-step guide with real-world examples.
Master parameterized datasets in ADF. Understand @dataset() vs @item(), avoid BadRequest errors, and build reusable datasets.
Master every ADF and Synapse expression. Understand @dataset(), @item(), @activity(), @pipeline() with real examples, scope rules, value flow diagrams, and a complete cheat sheet.
Build an incremental data loading pipeline in Azure Synapse using the watermark pattern. Copy only changed rows using config tables, delta queries, and stored procedures.
Build a production-grade unified pipeline that handles both full load and incremental load from a single config table. If Condition routes each table to the right strategy.
Decode every JSON structure in Azure Data Factory. Covers Linked Services, Datasets, Pipelines (simple copy, metadata-driven, incremental, audit logging), Triggers, Integration Runtimes, and Data Flows.
Why audit logging is essential for production pipelines. Design audit tables, stored procedures, and success/failure logging in ADF and Synapse.
Master SCD Types 0, 1, 2, 3, and 6 with detailed table examples and real-life analogies. Covers implementation in Azure SQL, Synapse Data Flows, and Delta Lake MERGE.
Build a complete SCD Type 1 pipeline from scratch with every activity explained.
Build an SCD Type 1 pipeline with SHA-256 hash-based change detection using Synapse Data Flows.
Build a complete SCD Type 2 pipeline using Synapse Data Flows. Every transformation explained: hash generation, lookup, conditional split, parallel expire+insert paths, dual sinks.
Build a production-grade pipeline combining data quality (null handling, standardization, deduplication) with dual SCD Type 1 and Type 2 using Synapse Data Flows.
Master Data Flows in ADF and Synapse. Covers all transformation types with real-life analogies, step-by-step build guide, Debug cluster costs, performance optimization, and when to use Data Flow vs Copy Activity.
Master every join type in Synapse Data Flows with real datasets and visual before/after tables. Covers Inner, Left Outer, Right Outer, Full Outer, Cross, Exists, and Lookup.
Master all three ADF trigger types. Schedule triggers for fixed-time runs, Tumbling Window for guaranteed time-series processing with backfill, and Event-Based triggers for file-driven pipelines.
Master trigger-to-pipeline parameter passing in ADF. Covers Schedule, Tumbling Window, and Storage Event trigger outputs, step-by-step wiring, system variables reference, real-world scenarios with analogies, and common…
Set up CI/CD for ADF and Synapse with GitHub. Complete guide covering Git integration, feature branches, ARM templates, GitHub Actions workflows, environment parameterization, and production deployment with manual…
Complete guide to CI/CD for ADF and Synapse using Azure DevOps. Covers Azure Repos setup, connecting ADF/Synapse to DevOps, YAML pipelines, multi-stage deployments, pre/post scripts, approval gates, variable groups…
CI/CD for ADF and Synapse: ARM templates, environment promotion, and deployment patterns.
Master all three Azure Integration Runtime types: Azure IR (auto-resolve and custom), Self-Hosted IR for on-premises access, and Azure-SSIS IR for SSIS packages.
Build a complete on-premises to Azure data pipeline using Self-Hosted Integration Runtime.
Build an automated cleanup pipeline in ADF that scans ADLS Gen2 folders, checks file ages using Get Metadata, and deletes files older than your retention period using If Condition and Delete activities.
Troubleshooting guide: common ADF and Synapse pipeline errors with solutions.