🧱
DATABRICKS
Databricks for Data Engineers
Delta Lake, Unity Catalog, Notebooks, Workflows, AutoLoader, Structured Streaming, Lakeflow, DABs, and production data engineering.
27 in-depth tutorials
Your first look at Databricks: workspace, clusters, notebooks, and essential dbutils commands.
Notebooks deep dive: cell types, magic commands, widgets, parameterization, and modular code patterns.
Connecting Databricks to Azure storage: Blob, ADLS Gen2, mount points, and credential passthrough.
Secret Scopes and Key Vault: secure credential management in Databricks.
Master reading and writing every file format in Databricks. Standard CSV, pipe-delimited, single-quote qualifiers, escape characters, multiline values, JSON, Parquet, and Delta Lake.
Delta Lake advanced: liquid clustering, deletion vectors, change data feed, and optimization.
Master Databricks to Azure SQL Database connectivity. JDBC connection setup, secure credentials with Key Vault, reading tables and custom queries, the ORDER BY subquery trap, write modes, upsert pattern, the…
08External Tables & Unity Catalog
Managed vs external tables: DROP behavior, storage lifecycle, and Unity Catalog integration.
09Delta Lake and PySpark Optimization
The complete optimization playbook for Databricks. Small file problem and OPTIMIZE compaction, Z-ORDER for file skipping, conditional OPTIMIZE, VACUUM with retention and time travel interaction, partitioning strategy…
10File Storage in Azure Databricks
File storage: Volumes, DBFS, /tmp/, external locations, and where files actually live.
11Data Quality in Azure Databricks
Build a production data quality framework in PySpark. Eight validation checks (nulls, duplicates, range, regex, referential integrity, freshness, row count, type), the quarantine pattern, a reusable DQ class, DQ…
12Databricks Workflows and Jobs
Workflows and Jobs: multi-task pipelines, scheduling, dependencies, and monitoring.
13Databricks Git Integration and CI/CD
Git integration and CI/CD: Repos, GitHub Actions, and deploying across environments.
14SCD Type 1 & Type 2 with PySpark Delta MERGE
Implement both SCD Type 1 and Type 2 using PySpark and Delta Lake MERGE in Databricks. Type 1: one MERGE with whenMatchedUpdate + whenNotMatchedInsertAll. Type 2: MERGE to expire + APPEND for new versions.
The complete AutoLoader guide for Databricks. cloudFiles source explained, why AutoLoader beats spark.read (7-feature comparison), file discovery modes (Directory Listing vs File Notification), reading CSV/JSON/Parquet…
Unity Catalog: metastore setup, three-level namespace, governance, lineage, and security.
17Unity Catalog Complete Reference
Every Unity Catalog object mapped: tables, views, volumes, functions, models, credentials, and Delta Sharing.
The complete guide to every meaning of managed and external in Databricks.
19Databricks SQL & SQL Warehouses
Databricks SQL: SQL Warehouses, serverless, dashboards, and query history.
Complete guide to streaming in Databricks. Batch vs streaming vs micro-batch, readStream and writeStream API, trigger modes (processingTime, availableNow), output modes (append, complete, update), checkpointing for…
21Databricks Asset Bundles (DABs)
Databricks Asset Bundles: YAML-based CI/CD, project structure, and deployment patterns.
22Lakeflow Declarative Pipelines
The complete Lakeflow Declarative Pipelines guide. Streaming tables for incremental ingestion. Materialized views for aggregations. Expectations for data quality. Medallion architecture. CDC with APPLY CHANGES.
The complete Lakeflow Connect guide. Managed SaaS connectors (Salesforce, HubSpot). Database connectors with CDC (SQL Server, PostgreSQL). Ingestion gateway. Incremental ingestion. Scheduling. Unity Catalog governance.
Advanced Delta Lake in Databricks: liquid clustering instead of partitioning, deletion vectors for faster updates and deletes, and UniForm for Iceberg-compatible reads. When and how to use each.
25Monitoring, Azure Monitor & AI/BI Genie
Monitoring Databricks: diagnostic logs, Azure Monitor, Spark UI, system tables, and AI/BI Genie.
DP-750 study guide: every exam objective mapped to blog posts with a study plan.
27Databricks DE Associate Study Guide
Every exam objective mapped across 7 domains, 6-week study plan, AutoLoader, Lakeflow Jobs, DABs, PySpark, Unity Catalog, and Delta Sharing.