Python

Python tutorials, tips, and best practices

PySpark DataFrame Transformations in Azure Databricks: The Complete Cookbook

Leave a Comment / Azure, Data Engineering, Python

The complete PySpark transformation cookbook for Databricks. Every function category with real code: column operations, filtering, withColumn, when/otherwise, string functions, date functions, null handling, aggregations (pivot, cube, rollup), window functions, joins, deduplication, complex types (arrays, structs, maps), nested JSON flattening, UDFs, and the pipeline pattern.

PySpark DataFrame Transformations in Azure Databricks: The Complete Cookbook Read More »

Reading and Writing Every File Format in Azure Databricks: CSV, Parquet, JSON, Delta, and Tricky CSV Variations

Leave a Comment / Azure, Data Engineering, Python

Master reading and writing every file format in Databricks. Standard CSV, pipe-delimited, single-quote qualifiers, escape characters, multiline values, JSON, Parquet, and Delta Lake. Covers all CSV options, writing with partitionBy, managed vs external tables, Delta operations, and a complete read-transform-write pipeline.

Reading and Writing Every File Format in Azure Databricks: CSV, Parquet, JSON, Delta, and Tricky CSV Variations Read More »

Azure Databricks for Data Engineers: Introduction, Architecture, and dbutils Commands Explained

Leave a Comment / Azure, Data Engineering, Python

Master Azure Databricks from architecture to daily commands. Covers workspace setup, cluster types, notebooks, and every dbutils module: fs (file operations), secrets (Key Vault integration), widgets (parameterization), and notebook (orchestration). Plus Delta Lake operations, mounting storage, Workflows, cost management, and Databricks vs Synapse comparison.

Azure Databricks for Data Engineers: Introduction, Architecture, and dbutils Commands Explained Read More »

Apache Spark and PySpark for Data Engineers: Architecture, Python vs PySpark, and Big Data Processing

Leave a Comment / Data Engineering, Python

Master Apache Spark and PySpark from architecture to code. Covers Driver-Executor model, lazy evaluation, RDDs vs DataFrames, Python vs PySpark comparison with code examples, all DataFrame operations, Spark SQL, partitioning, shuffling, broadcast joins, window functions, performance tuning, and Azure integration.

Apache Spark and PySpark for Data Engineers: Architecture, Python vs PySpark, and Big Data Processing Read More »

Fine-Tuning Large Language Models: A Complete Guide for Data Engineers

Leave a Comment / AI & Machine Learning, Data Engineering, Python

Master LLM fine-tuning from concepts to code. Covers when to fine-tune vs RAG vs prompt engineering, LoRA and QLoRA methods, step-by-step with OpenAI API and Hugging Face, training data preparation, 5 real-world scenarios, evaluation techniques, costs, and the data engineer role in AI projects.

Fine-Tuning Large Language Models: A Complete Guide for Data Engineers Read More »

Scroll to Top