Python tutorials, tips, and best practices
The complete Python data validation guide. pydantic for record-level schema enforcement. pandera for DataFrame validation. Great Expectations for enterprise data quality. Schema checks, custom validators, data contracts. Eight mistakes and seven Q&As.
The complete Python environment management guide. venv and pip basics. pip-tools for lockfiles. conda for native libraries. Poetry for project management. uv as the 2026 default. Docker for production. Decision framework. Eight mistakes and seven Q&As.
The complete pandas deep dive for data engineers. groupby with agg, transform, and filter. Named aggregations. merge and join for combining DataFrames (inner, left, right, outer, cross). concat for stacking. pivot_table for wide format with aggregation. melt for long format. stack and unstack for multi-index reshaping. apply for row-wise and column-wise custom functions. pipe for clean method chaining. Real-world data engineering patterns. Eight common mistakes and seven interview Q&As.
The complete Python packaging and CLI tools guide for data engineers. argparse for quick command-line scripts. click for professional CLIs with subcommands. pyproject.toml as the modern standard for project configuration. setuptools for building packages. Entry points for creating installable commands. Project structure with src layout. pip install -e for development mode. Building and distributing wheels. Real-world CLI tools for ETL pipelines. Eight common mistakes and seven interview Q&As.
The complete Python cloud storage guide for data engineers. boto3 for AWS S3: client vs resource, upload, download, list with pagination, read directly into pandas, presigned URLs, multipart transfers. azure-storage-blob for Azure Blob Storage and ADLS Gen2: BlobServiceClient, ContainerClient, BlobClient, DefaultAzureCredential, connection strings, SAS tokens. google-cloud-storage for GCS. Cross-cloud comparison table. Production patterns: cloud-to-local ETL, cloud-to-cloud copy, partitioned uploads. Eight common mistakes and seven interview Q&As.
The complete Python concurrency guide for data engineers. The GIL and why it matters. CPU-bound vs I/O-bound tasks. threading for concurrent I/O. multiprocessing for true parallelism. concurrent.futures with ThreadPoolExecutor and ProcessPoolExecutor. asyncio fundamentals: async, await, event loop, gather, create_task. aiohttp for parallel API calls. Decision framework for choosing the right approach. Real-world patterns: parallel file processing, concurrent API extraction, parallel DataFrame transforms. Eight common mistakes and seven interview Q&As.
The complete pytest guide for data engineers. Why testing matters for data pipelines. pytest basics and plain assert. Fixtures, fixture scope, yield fixtures, and conftest.py. Parametrize for testing multiple inputs. Markers for categorizing tests. Mocking with unittest.mock, patch, and monkeypatch. Testing ETL transform functions, database code with sqlite3, and API extractors with mocked responses. tmp_path for file-based tests. Coverage reporting. Arrange-Act-Assert pattern. Project structure. Eight common mistakes and seven interview Q&As.
The complete Python ETL patterns guide for data engineers. ETL vs ELT and when to use each. Building extract, transform, and load phases with pandas. Function-based and class-based pipeline architectures. Error handling with retry and dead letter patterns. Structured logging for pipelines. Configuration-driven pipelines. Incremental vs full load with watermark tables. Data validation before and after transforms. Idempotent loads. End-to-end real-world pipeline. Eight common mistakes and seven interview Q&As.
The complete Python database connections guide for data engineers. SQLAlchemy 2.0 engines, connection strings, and the text() function. pyodbc for SQL Server and Azure SQL. psycopg2 for PostgreSQL. sqlite3 for local development. pandas read_sql() and to_sql() for DataFrame I/O. Connection pooling, parameterized queries, transactions, environment variables for secrets, Azure SQL with Managed Identity, and production patterns. Eight common mistakes and seven interview Q&As.
The complete Python file formats guide for data engineers. Reading and writing CSV files with every pandas parameter explained. JSON flat, nested, and JSON Lines patterns. Parquet with pyarrow — columnar storage, compression, partitioning, and predicate pushdown. Excel with openpyxl — multiple sheets, formatting, and headers. Converting between formats. Handling large files with chunking. Arrow-backed dtypes for memory efficiency. Eight real-world patterns, common mistakes, and interview Q&As.
