Azure Data Factory, Synapse, pipelines
The complete Lakeflow Declarative Pipelines guide. Streaming tables for incremental ingestion. Materialized views for aggregations. Expectations for data quality. Medallion architecture. CDC with APPLY CHANGES. Triggered vs continuous modes. SQL and Python syntax. Eight mistakes and seven Q&As.
The complete Python cloud storage guide for data engineers. boto3 for AWS S3: client vs resource, upload, download, list with pagination, read directly into pandas, presigned URLs, multipart transfers. azure-storage-blob for Azure Blob Storage and ADLS Gen2: BlobServiceClient, ContainerClient, BlobClient, DefaultAzureCredential, connection strings, SAS tokens. google-cloud-storage for GCS. Cross-cloud comparison table. Production patterns: cloud-to-local ETL, cloud-to-cloud copy, partitioned uploads. Eight common mistakes and seven interview Q&As.
The complete KQL window functions reference. The serialize operator and why window functions require it. prev() and next() for accessing adjacent rows. row_number() for sequential numbering. row_cumsum() for running totals. row_rank_dense() and row_rank_min() for ranking (dense vs gaps). row_window_session() for automatic session detection. The scan operator for stateful row processing. Partitioned windows with the restart parameter. KQL vs SQL window function comparison table. Eight real-world patterns, common mistakes, and interview Q&As.
The complete guide to the Microsoft Fabric Capacity Metrics App. What Capacity Units (CUs) are and how they are consumed. How to install the app from AppSource. Every page explained: Health, Compute, Storage, Timepoint, Timepoint Summary, Timepoint Item Detail, Item History, Autoscale Compute for Spark. Interactive vs background operations and how smoothing works. Throttling stages (delay, rejection) and how to diagnose them. The Chargeback App for cost allocation. Real-world diagnostic scenarios, eight common mistakes, and seven interview Q&As.
The complete DP-700 study guide with every exam objective mapped to 38 DriveDataScience Fabric posts. Three domains: Implement and Manage, Ingest and Transform, Monitor and Optimize. 8-week study plan, quick reference cards, exam-day tips, and 5 practice questions.
Complete Fabric Data Factory expression language guide. Where expressions work (Copy, Notebook, If Condition, ForEach, Web activities). Pipeline functions (@pipeline, @activity, @variables, @item). String functions (concat, replace, split, trim). Date functions (utcNow, formatDateTime, addDays, startOfMonth). Logical functions (if, equals, coalesce, and, or). Eight real-world patterns: yesterday incremental load, dynamic file paths with date partitioning, conditional full vs incremental, dynamic SQL, ForEach table list, notebook parameters, error handling, dynamic email subjects. ADF vs Fabric comparison (identical syntax).
The complete KQL reference guide. Pipe-based syntax explained with SQL comparison. Filtering with where (comparison, string operators, has vs contains performance). Selecting with project and extend. Aggregation with summarize (count, dcount, percentile, arg_max, make_list). Time-based analysis with bin and ago. All join types including lookup optimization. Complete function reference: string, date/time, numeric, dynamic/JSON. Let statements for variables and subqueries. Render for inline visualization. Advanced: make-series for time series, series_decompose_anomalies for anomaly detection, materialized views, stored functions. Eight real-world query patterns (latest per device, sessions, error rate, top N, funnel, spike detection, week-over-week, distributed tracing). Full KQL vs SQL comparison table.
The complete M language reference for Dataflow Gen2. The let-in structure explained. Every essential function: text (20+ functions including split, combine, replace, extract), numbers (round, math, conversion), dates (30+ functions including arithmetic, extraction, start/end of periods, formatting), logical (if-then-else, null coalescing), lists (aggregate, filter, transform), and tables (filter rows, add columns, join with all 6 join kinds, group by, pivot, unpivot, buffer). Error handling with try-otherwise. Custom functions. Query folding explained. Five real-world patterns.
Complete Fabric connections and gateways guide. Connection types for Azure, on-premises, cross-cloud (AWS S3, GCS), and SaaS sources with authentication options. Connection vs ADF Linked Service migration. On-premises data gateway installation, architecture (outbound HTTPS only, no inbound ports), and high-availability clustering. VNet data gateway for private Azure resources. Creating, sharing, and reusing connections. Security best practices (Service Principal, Key Vault). Troubleshooting connection errors. Three real-world scenarios.
The complete Delta Lake Table Properties reference. Every TBLPROPERTIES setting explained: deletedFileRetentionDuration and logRetentionDuration with VACUUM interaction, Change Data Feed (CDF) for MLVs and streaming, column mapping for rename and drop columns, autoOptimize.optimizeWrite and autoCompact, targetFileSize, protocol versions, schema.autoMerge, and data skipping. Five real-world configurations (Gold SCD dimension, Bronze staging, Silver standard, streaming target, large fact table). Fabric vs Databricks property comparison.
