Complete Fabric Spark configuration and performance tuning guide. Where to set configurations (Environments for persistent, %%configure for session-level, spark.conf.set for runtime) with mutable vs immutable properties table and the critical root-vs-conf placement rule for %%configure. The Big Three settings: spark.sql.shuffle.partitions (default 200, grocery checkout lane analogy, guidelines by data size, demonstration with empty partitions), spark.sql.autoBroadcastJoinThreshold (default 10MB, product catalog analogy, why increase to 100-256MB, manual broadcast hint, OOM warning with 20% rule, checking join strategy with explain()), and spark.sql.files.maxPartitionBytes (default 128MB, delivery truck analogy, when to increase/decrease). Adaptive Query Execution with GPS analogy (coalesce partitions, auto broadcast conversion, skew join splitting, AQE vs manual tuning). Autotune ML-based optimizer (how to enable, when it works best, checking driver log recommendations). Native Execution Engine (Velox/Gluten C++ engine, 2-4x faster, enable via %%configure or Environment, UDF fallback limitation). Memory and resources (driver vs executor memory, node sizes table, starter vs custom pools comparison, high concurrency mode). Delta Lake settings (optimize write, auto compaction, V-Order default-on, target file size). Join strategy deep dive (broadcast hash, sort-merge, shuffle hash with decision table). Four production configuration templates (small/medium/large/massive data). Spark UI diagnosis (reading jobs/stages/executors tabs, identifying shuffle bottlenecks, identifying data skew with median vs max gap). 8 common mistakes and 8 interview Q&As.
