Azure - DriveDataScience

Dataflow Gen2 in Production: Pipeline Integration, Parameterization, Incremental Refresh, Performance Optimization, and the Complete Decision Guide

Leave a Comment / Azure, Data Engineering

Take Dataflow Gen2 to production. Pipeline integration patterns (Copy then Dataflow then Notebook then Refresh), parameterization (create, use in filters, pass from pipeline), incremental refresh with date filters and watermark tables, query folding explained with foldable vs non-foldable steps table, performance optimization (reduce at source, column selection, buffering), monitoring and debugging, the complete Dataflow Gen2 vs Notebook decision matrix with 20 scenarios, Medallion Architecture mapping, and three real-world production examples.

Dataflow Gen2 in Production: Pipeline Integration, Parameterization, Incremental Refresh, Performance Optimization, and the Complete Decision Guide Read More »

Dataflow Gen2 Advanced Transformations: Merge Queries, Append, Pivot, Group By, Custom Columns, and Error Handling

Leave a Comment / Azure, Data Engineering

Master Dataflow Gen2 advanced transformations. Merge Queries with all 6 join types and fuzzy matching, Append Queries for UNION ALL, Group By with multiple aggregations, Pivot and Unpivot (with Unpivot Other Columns best practice), Conditional Columns as no-code CASE WHEN, Custom Columns with 25+ M formula examples (string, date, null handling, conditional), Replace Errors and try-otherwise pattern, Data Profiling (column quality, distribution, profile), complete 9-step Bronze-to-Silver example, and when Dataflow Gen2 reaches its limits.

Dataflow Gen2 Advanced Transformations: Merge Queries, Append, Pivot, Group By, Custom Columns, and Error Handling Read More »

Dataflow Gen2 in Microsoft Fabric: Introduction, Power Query Basics, Connecting to Sources, and Your First No-Code ETL

Leave a Comment / Azure, Data Engineering

The complete Dataflow Gen2 introduction. What it is vs ADF Mapping Data Flows vs Spark Notebooks, Power Query engine and M language explained, the UI walkthrough with three panels, connecting to all source types (Lakehouse, SQL, CSV, SharePoint), every basic transformation step-by-step (Choose Columns, Filter, Rename, Change Type, Replace Values, Add Column from Examples, Trim, Split, Fill Down, Remove Duplicates, Sort), writing to Lakehouse and Warehouse destinations with Replace vs Append update methods, and monitoring runs.

Dataflow Gen2 in Microsoft Fabric: Introduction, Power Query Basics, Connecting to Sources, and Your First No-Code ETL Read More »

Lakehouse vs Warehouse in Microsoft Fabric: When to Use Which, What Languages Work Where, and Real-World Scenario Guide

Leave a Comment / Azure, Data Engineering

The definitive Lakehouse vs Warehouse guide for Microsoft Fabric. Side-by-side comparison across 17 features, the SQL analytics endpoint explained (why read-only), languages and interfaces matrix (PySpark, SparkSQL, T-SQL — what works where), read vs write capabilities table, security model differences, five real-world scenarios (e-commerce ETL, financial reporting, IoT, Customer 360 with ML, self-service analytics), the recommended Medallion pattern (Lakehouse for Bronze/Silver, Warehouse for Gold), cross-database queries, and migration guide from Synapse/Databricks.

Lakehouse vs Warehouse in Microsoft Fabric: When to Use Which, What Languages Work Where, and Real-World Scenario Guide Read More »

Fabric Data Factory: Activities, Pipelines, Dataflow Gen2, Notebooks, and Building Production ETL in Microsoft Fabric

Leave a Comment / Azure, Data Engineering

The complete Fabric Data Factory guide. What changed from ADF (no datasets, connections instead of linked services, Dataflow Gen2 instead of Mapping Data Flows). All pipeline activities listed: data movement, transformation, control flow, notification (Teams, Outlook — NEW), and Fabric-specific (Semantic Model Refresh). Three complete pipeline examples including metadata-driven load and full Medallion ETL combining Copy + Dataflow Gen2 + Notebook + Power BI Refresh + Teams notification.

Fabric Data Factory: Activities, Pipelines, Dataflow Gen2, Notebooks, and Building Production ETL in Microsoft Fabric Read More »

OneLake Shortcuts in Microsoft Fabric: Every Source, Every Permission, and How to Access Data Without Copying It

Leave a Comment / Azure, Data Engineering

Master OneLake shortcuts in Fabric. Every supported source (ADLS Gen2, S3, S3-compatible, GCS, Dataverse, on-premises, Iceberg), read/write/delete behavior per source, the delete trap explained, two-layer security model, authentication methods per source, shortcut caching for cross-cloud cost savings, chained shortcuts, Direct Lake with shortcuts for Power BI, trusted workspace access for private ADLS, four real-world patterns, and step-by-step creation guide.

OneLake Shortcuts in Microsoft Fabric: Every Source, Every Permission, and How to Access Data Without Copying It Read More »

Microsoft Fabric Foundations: Capacity, Workspaces, Items, OneLake, and the Building Blocks Every Data Engineer Must Understand

Leave a Comment / Azure, Data Engineering

Master the building blocks of Microsoft Fabric. Capacity explained with the apartment building analogy, all F-SKU options with pricing, PAYG vs Reserved, pause/resume cost savings, the F64 threshold, workspaces and roles, all Fabric items listed and explained, Lakehouse vs Warehouse decision guide, OneLake storage and shortcuts, environment setup patterns, and the free 60-day trial.

Microsoft Fabric Foundations: Capacity, Workspaces, Items, OneLake, and the Building Blocks Every Data Engineer Must Understand Read More »

Azure Connections and Authentication for Data Engineers: Every Service, Every Method, and How to Remember Them All

Leave a Comment / Azure, Data Engineering

The Azure connections reference card for data engineers. Five authentication methods explained with building key analogies (master key, visitor badge, facial recognition, employee badge, full address). Every service covered: ADLS, SQL, Key Vault, Databricks, ADF, Fabric, Event Hubs, Power BI. Complete connection matrix, endpoint formats, connection strings, secure vs quick decision table, troubleshooting guide, and one-page cheat sheet.

Azure Connections and Authentication for Data Engineers: Every Service, Every Method, and How to Remember Them All Read More »

Microsoft Fabric for Data Engineers: What It Is, What It Replaces, How It Competes, and Why It Matters

Leave a Comment / Azure, Data Engineering

The complete guide to Microsoft Fabric for data engineers. What it is, all 7 workloads explained, OneLake as the universal storage layer, what Azure services it replaces (13-row mapping table), how our blog pipelines translate to Fabric, head-to-head comparisons with Databricks and Snowflake and AWS, Direct Lake mode for Power BI, the DP-700 certification, capacity-based pricing, migration path, and when to use Fabric vs Databricks vs both.

Microsoft Fabric for Data Engineers: What It Is, What It Replaces, How It Competes, and Why It Matters Read More »

How Real Companies Receive Data: SFTP, APIs, CDC, Event Streaming, and Every Ingestion Pattern Explained

Leave a Comment / Azure, Data Engineering

How data actually arrives in production — not from tutorials, from real companies. Six ingestion patterns: SFTP file drops, REST API pulls, CDC database replication, event streaming, direct cloud drops, and third-party tools. Complete architectures for banking, e-commerce, telecom, healthcare, retail, and insurance with exact data flow diagrams.

How Real Companies Receive Data: SFTP, APIs, CDC, Event Streaming, and Every Ingestion Pattern Explained Read More »