Python Modules, Packages, Virtual Environments, and Imports: How Python Organizes Code, Resolves Dependencies, and Every Pattern Data Engineers Need

Your pipeline started as one file — pipeline.py. 200 lines. Easy to read. Then you added validation. Then logging. Then retry logic. Then config loading. Then five more tables. Now it is 2,000 lines and nobody wants to touch it. You cannot reuse the retry logic in another project because it is buried inside a giant file. You cannot test the validation without running the entire pipeline. You added a library for API calls, but it conflicts with a library another project uses.

Modules, packages, and virtual environments solve all of this. Modules let you split code into separate files. Packages let you organize files into directories. Virtual environments let you isolate dependencies per project. Imports let you connect everything together.

Real-life analogy: Think of your codebase like a kitchen. A single-file script is like cooking everything on one counter — ingredients, utensils, cutting board, and stove all in one pile. A module is like organizing into stations — prep station, cooking station, plating station. A package is like separate rooms — the bakery, the grill kitchen, the cold kitchen. Each room has its own stations (modules). Virtual environments are like separate kitchens for separate restaurants — Restaurant A uses French recipes (library v1.0) and Restaurant B uses Italian recipes (library v2.0). They share the same building but never mix ingredients. And imports are like saying “bring me the knife from the prep station” — you specify exactly what you need from where.

Table of Contents

• Modules — Splitting Code into Files
• What Is a Module
• Creating Your First Module
• Import Styles (import, from…import, as)
• What Happens When You Import
• The __name__ == “__main__” Guard
• Module Search Path (sys.path)
• Packages — Organizing Modules into Directories
• What Is a Package
• Creating a Package
• __init__.py — The Package Initializer
• Nested Packages (Sub-packages)
• Relative vs Absolute Imports
• Import Best Practices
• Import Order (PEP 8)
• Circular Imports and How to Fix Them
• Lazy Imports
• Star Imports and Why to Avoid Them
• The Standard Library — Batteries Included
• Essential Standard Library Modules for Data Engineers
• Third-Party Packages (pip)
• Installing with pip
• requirements.txt
• Pinning Versions
• Finding Packages (PyPI)
• Virtual Environments
• Why Virtual Environments
• Creating and Using venv
• Activating and Deactivating
• Installing Packages in a Virtual Environment
• requirements.txt Workflow
• When to Create a New Virtual Environment
• Other Tools (conda, poetry, pipenv)
• Data Engineering Patterns
• Project Structure for a Data Pipeline
• Config Module Pattern
• Utils Module Pattern
• Pipeline Package with Extract/Transform/Load Modules
• Shared Library Across Multiple Projects
• Common Mistakes
• Interview Questions
• Wrapping Up

Modules — Splitting Code into Files

What Is a Module

A module is simply a .py file. Any Python file is a module. When you write import math, you are importing the math.py module from the standard library. When you write import helpers, you are importing your own helpers.py file. A module can contain functions, classes, variables, and executable code.

Creating Your First Module

# FILE: utils.py — your first module
"""Utility functions for data pipelines."""

import logging
from datetime import datetime

logger = logging.getLogger(__name__)

def clean_string(value):
    """Strip whitespace and convert to lowercase."""
    if value is None:
        return None
    return str(value).strip().lower()

def parse_date(date_str, fmt="%Y-%m-%d"):
    """Parse a date string into a datetime object."""
    try:
        return datetime.strptime(date_str, fmt)
    except (ValueError, TypeError):
        logger.warning(f"Could not parse date: {date_str}")
        return None

def chunked(iterable, size):
    """Split an iterable into chunks of the given size."""
    for i in range(0, len(iterable), size):
        yield iterable[i:i + size]

# Constants
DEFAULT_BATCH_SIZE = 10000
VALID_ENVIRONMENTS = {"development", "staging", "production"}

# FILE: pipeline.py — uses the utils module
import utils

# Use functions from the module
name = utils.clean_string("  NAVEEN  ")
print(name)   # "naveen"

date = utils.parse_date("2026-07-08")
print(date)   # 2026-07-08 00:00:00

# Use constants from the module
print(utils.DEFAULT_BATCH_SIZE)      # 10000
print(utils.VALID_ENVIRONMENTS)      # {"development", "staging", "production"}

# Process data in chunks
records = list(range(100))
for chunk in utils.chunked(records, 25):
    print(f"Processing batch of {len(chunk)} records")

Import Styles (import, from…import, as)

# Style 1: import module — access with module.function()
import utils
utils.clean_string("hello")
utils.parse_date("2026-07-08")
# PRO: Clear where the function comes from (utils.clean_string)
# CON: More typing

# Style 2: from module import specific_items
from utils import clean_string, parse_date
clean_string("hello")         # No prefix needed
parse_date("2026-07-08")
# PRO: Less typing, cleaner code
# CON: Less clear where functions come from in large files

# Style 3: from module import * — import EVERYTHING
from utils import *
clean_string("hello")
DEFAULT_BATCH_SIZE
# ❌ AVOID: pollutes namespace, hides where things come from, causes name conflicts

# Style 4: import with alias (as)
import pandas as pd
import numpy as np
from datetime import datetime as dt
import utils as u
u.clean_string("hello")
# PRO: Shorter names for frequently used modules
# Standard aliases: pd (pandas), np (numpy), plt (matplotlib.pyplot)

# Style 5: from package.module import specific_item
from pathlib import Path
from collections import defaultdict, Counter
from typing import List, Dict, Optional

Real-life analogy: import utils is like saying “bring me the toolbox” — you access tools as toolbox.hammer, toolbox.screwdriver. from utils import clean_string is like saying “bring me just the hammer” — you use it directly as hammer. from utils import * is like dumping the entire toolbox on the table — fast, but now you cannot tell which tools came from which box, and if two boxes have a “hammer,” you do not know which one you are using.

What Happens When You Import

# When you write "import utils", Python does THREE things:

# 1. FINDS the module — searches sys.path (current directory, then installed packages)
# 2. EXECUTES the module — runs ALL the code in utils.py from top to bottom
# 3. CREATES a module object — stores everything (functions, classes, variables)
#    in a namespace accessible as utils.function_name

# IMPORTANT: The module is executed ONLY ONCE, even if you import it multiple times
import utils      # First import: finds, executes, creates module object
import utils      # Second import: returns the SAME cached module object (no re-execution)

# The cached module is stored in sys.modules
import sys
print("utils" in sys.modules)   # True after first import

# This is why module-level code (like print statements) runs only once:
# FILE: noisy_module.py
# print("I was imported!")       # Runs ONCE, even if imported 10 times

# And why module-level variables persist:
# utils.counter = 0              # Set once during import
# utils.counter += 1             # Modifying it — the change persists across all imports

The __name__ == “__main__” Guard

# Every module has a __name__ attribute
# When you RUN a file directly: __name__ = "__main__"
# When you IMPORT a file: __name__ = "module_name" (e.g., "utils")

# FILE: utils.py
def clean_string(value):
    return str(value).strip().lower()

def parse_date(date_str):
    from datetime import datetime
    return datetime.strptime(date_str, "%Y-%m-%d")

# This code ONLY runs when utils.py is executed directly (python utils.py)
# It does NOT run when another file does "import utils"
if __name__ == "__main__":
    # Test the functions
    print(clean_string("  HELLO  "))       # "hello"
    print(parse_date("2026-07-08"))        # 2026-07-08 00:00:00
    print("All tests passed!")

# Why this matters:
# Without the guard, "import utils" would print test output every time
# With the guard, "import utils" silently loads the functions
# Running "python utils.py" directly shows the test output

# The guard is also used to make scripts that are BOTH importable and executable:
# - As a module: from utils import clean_string (test code does not run)
# - As a script: python utils.py (test code runs)

Real-life analogy: The __name__ == "__main__" guard is like a “For Display Only” tag at a furniture store. When you visit the store (import the module), the furniture is just for looking — you do not sit in the chairs or eat at the tables. But when you live in the house (run the file directly), you use everything normally. The guard separates “display mode” (imported) from “live mode” (executed directly).

Module Search Path (sys.path)

import sys

# When you write "import utils", Python searches these locations IN ORDER:
for i, path in enumerate(sys.path):
    print(f"{i}: {path}")

# Typical order:
# 0: "" (current directory — where the script is running)
# 1: /usr/lib/python3.11                   (standard library)
# 2: /usr/lib/python3.11/lib-dynload       (C extensions)
# 3: /home/naveen/venv/lib/python3.11/site-packages  (installed packages)

# Python uses the FIRST match. If you have a file named "json.py" in your
# current directory, it SHADOWS the standard library json module!
# ❌ NEVER name your files: json.py, csv.py, math.py, os.py, logging.py, etc.

# Add a custom directory to the search path (temporary, for this session only):
sys.path.insert(0, "/path/to/my/shared/modules")

# Permanent: set the PYTHONPATH environment variable
# export PYTHONPATH="/path/to/my/shared/modules:$PYTHONPATH"

Packages — Organizing Modules into Directories

What Is a Package

A package is a directory containing Python modules and a special __init__.py file. Packages let you organize related modules into a folder hierarchy — just like a filing cabinet has drawers (packages) that contain folders (sub-packages) that contain documents (modules).

Creating a Package

# Directory structure:
# pipeline/
#     __init__.py        ← Makes "pipeline" a package (can be empty)
#     extract.py         ← Module for extraction logic
#     transform.py       ← Module for transformation logic
#     load.py            ← Module for loading logic
#     utils.py           ← Shared utility functions

# FILE: pipeline/extract.py
import logging
logger = logging.getLogger(__name__)   # logger name: "pipeline.extract"

def extract_from_sql(connection_string, table_name):
    logger.info(f"Extracting from SQL: {table_name}")
    # ... extraction logic
    return data

def extract_from_api(url):
    logger.info(f"Extracting from API: {url}")
    # ... API call
    return data


# FILE: pipeline/transform.py
def clean_nulls(data, columns):
    """Replace None with defaults."""
    for row in data:
        for col in columns:
            if row.get(col) is None:
                row[col] = ""
    return data

def add_audit_columns(data):
    """Add load_date and source columns."""
    from datetime import datetime
    now = datetime.now().isoformat()
    for row in data:
        row["_loaded_at"] = now
    return data


# FILE: pipeline/load.py
def load_to_warehouse(data, target_table):
    print(f"Loading {len(data)} rows to {target_table}")
    # ... load logic


# USING the package from outside:
# FILE: main.py (in the parent directory of pipeline/)
from pipeline.extract import extract_from_sql
from pipeline.transform import clean_nulls, add_audit_columns
from pipeline.load import load_to_warehouse

data = extract_from_sql("server.db.windows.net", "customers")
data = clean_nulls(data, ["email", "phone"])
data = add_audit_columns(data)
load_to_warehouse(data, "silver.customers")

__init__.py — The Package Initializer

# __init__.py runs when the package is first imported
# It can be empty (just marks the directory as a package)
# Or it can expose a clean public API:

# FILE: pipeline/__init__.py

# Option 1: Empty file — users import from specific modules
# from pipeline.extract import extract_from_sql
# from pipeline.transform import clean_nulls

# Option 2: Re-export commonly used items for convenience
from pipeline.extract import extract_from_sql, extract_from_api
from pipeline.transform import clean_nulls, add_audit_columns
from pipeline.load import load_to_warehouse

# Now users can do:
# from pipeline import extract_from_sql, load_to_warehouse
# Instead of:
# from pipeline.extract import extract_from_sql
# from pipeline.load import load_to_warehouse

# Option 3: Define __all__ to control what "from pipeline import *" exports
__all__ = [
    "extract_from_sql",
    "extract_from_api",
    "clean_nulls",
    "add_audit_columns",
    "load_to_warehouse",
]

# Option 4: Package-level configuration
import logging
logging.getLogger(__name__).addHandler(logging.NullHandler())

__version__ = "1.0.0"

Real-life analogy: __init__.py is like the reception desk of an office building. When someone visits (imports), the reception desk tells them what is available and how to access it. An empty __init__.py is like a building with no reception — visitors have to find rooms themselves (from pipeline.extract import extract_from_sql). A well-crafted __init__.py is like a reception that says “what do you need? I can get it for you” — simpler access for the visitor.

Nested Packages (Sub-packages)

# Real-world project structure with nested packages:
# pipeline/
#     __init__.py
#     config.py
#     sources/
#         __init__.py
#         sql.py
#         api.py
#         file.py
#     transforms/
#         __init__.py
#         cleaning.py
#         enrichment.py
#         validation.py
#     sinks/
#         __init__.py
#         warehouse.py
#         delta.py
#         parquet.py

# Usage:
from pipeline.sources.sql import SQLExtractor
from pipeline.sources.api import APIExtractor
from pipeline.transforms.cleaning import remove_nulls
from pipeline.transforms.validation import validate_schema
from pipeline.sinks.warehouse import WarehouseLoader
from pipeline.sinks.delta import DeltaWriter

# Each sub-package has its own __init__.py — each directory needs one

Relative vs Absolute Imports

# ABSOLUTE imports — always use the full path from the project root
# Recommended by PEP 8 — unambiguous and clear
from pipeline.extract import extract_from_sql
from pipeline.transform import clean_nulls
from pipeline.utils import chunked

# RELATIVE imports — use dots to refer to the current package
# Only work INSIDE a package (not in top-level scripts)
# FILE: pipeline/transform.py
from .utils import chunked          # . = current package (pipeline)
from .extract import extract_from_sql   # . = sibling module
from ..shared.logging import setup_logger  # .. = parent package

# One dot (.) = current package
# Two dots (..) = parent package
# Three dots (...) = grandparent package

# RECOMMENDATION: Use absolute imports everywhere.
# They are clearer, work in all contexts, and are easier to debug.
# Reserve relative imports for internal package modules only.

Import Best Practices

Import Order (PEP 8)

# PEP 8 specifies a strict import order with blank lines between groups:

# GROUP 1: Standard library imports
import os
import sys
import json
import logging
from pathlib import Path
from datetime import datetime, timedelta
from collections import defaultdict

# GROUP 2: Third-party library imports
import pandas as pd
import requests
from pyspark.sql import SparkSession
from azure.storage.blob import BlobServiceClient

# GROUP 3: Local application imports
from pipeline.extract import extract_from_sql
from pipeline.transform import clean_nulls
from pipeline.config import PipelineConfig
from utils import chunked, retry

# Within each group, sort alphabetically
# Each group separated by ONE blank line
# This makes it easy to scan imports and spot missing dependencies

Circular Imports and How to Fix Them

# CIRCULAR IMPORT: A imports B, and B imports A → ImportError

# ❌ Circular — breaks
# FILE: extract.py
from transform import clean_data     # extract needs transform

# FILE: transform.py
from extract import get_schema       # transform needs extract → CIRCULAR!

# FIX 1: Move the import inside the function (lazy import)
# FILE: transform.py
def transform_data(data):
    from extract import get_schema    # Import ONLY when function is called
    schema = get_schema()
    # ... transform logic

# FIX 2: Restructure — move shared code to a third module
# FILE: schema.py
def get_schema():
    return {"id": "int", "name": "str"}

# FILE: extract.py
from schema import get_schema

# FILE: transform.py
from schema import get_schema

# FIX 3: Combine the modules if they are tightly coupled
# Sometimes two modules that import each other should be one module

# RULE: If you have a circular import, it is a sign of bad structure.
# Fix the structure, do not patch the import.

Lazy Imports

# Import a module INSIDE a function instead of at the top of the file
# Use when the module is expensive to import, optional, or rarely used

def export_to_excel(data, filepath):
    """Only import openpyxl when actually exporting to Excel."""
    import openpyxl  # Heavy library — only loaded if this function is called
    wb = openpyxl.Workbook()
    ws = wb.active
    for row in data:
        ws.append(row)
    wb.save(filepath)

def detect_encoding(filepath):
    """Only import chardet when needed."""
    try:
        import chardet   # Optional dependency — may not be installed
    except ImportError:
        raise ImportError("chardet is required for encoding detection. pip install chardet")

    with open(filepath, "rb") as f:
        result = chardet.detect(f.read(10000))
    return result["encoding"]

# Benefits of lazy imports:
# 1. Faster startup — heavy libraries are not loaded until needed
# 2. Optional dependencies — code works even if the library is not installed
# 3. Avoids circular imports — delays the import until runtime

Star Imports and Why to Avoid Them

# ❌ Star imports — import everything from a module
from utils import *

# Why this is dangerous:
# 1. You cannot tell where functions come from
clean_string("hello")    # Is this from utils? From another module? Built-in?

# 2. Name collisions — silently overwrite existing functions
from module_a import *   # Defines "process()"
from module_b import *   # Also defines "process()" — overwrites module_a's version!
process()                # Which one? module_b's — no warning

# 3. No IDE support — auto-complete and linting cannot help you
# 4. Makes code review harder — reviewers cannot see what is being used

# ✅ Import only what you need
from utils import clean_string, parse_date, DEFAULT_BATCH_SIZE

# EXCEPTION: In __init__.py, star imports are acceptable to re-export
# from .extract import *   — OK in __init__.py to build the public API

The Standard Library — Batteries Included

Essential Standard Library Modules for Data Engineers

Module	What It Does	Example Use
os	Operating system interface	Environment variables, file paths
sys	System-level operations	Command-line args, module path
pathlib	Object-oriented file paths	Build paths, list files, check existence
json	JSON parsing and writing	Config files, API responses
csv	CSV reading and writing	DictReader, DictWriter for data files
logging	Logging framework	Structured pipeline logging
datetime	Date and time operations	Timestamps, date math, formatting
collections	Specialized containers	defaultdict, Counter, namedtuple, deque
typing	Type hints	List[str], Dict[str, int], Optional[str]
functools	Higher-order functions	lru_cache, partial, reduce
itertools	Iterator utilities	chain, groupby, islice, product
re	Regular expressions	Pattern matching, text extraction
hashlib	Cryptographic hashing	MD5/SHA256 checksums for data integrity
tempfile	Temporary files and dirs	Safe temp files for atomic writes
shutil	File operations	Copy, move, delete files and directories
unittest	Testing framework	Unit tests for pipeline functions
argparse	Command-line argument parsing	Pipeline CLI with –table, –env flags
configparser	INI file parsing	Legacy config file reading
abc	Abstract base classes	Enforce interfaces for pipeline components
concurrent.futures	Parallel execution	ThreadPoolExecutor for concurrent API calls

# Quick examples of essential standard library usage:

# collections — defaultdict, Counter
from collections import defaultdict, Counter
table_counts = defaultdict(int)
table_counts["customers"] += 50000
table_counts["orders"] += 100000

status_counts = Counter(["success", "success", "failed", "success"])
print(status_counts)  # Counter({'success': 3, 'failed': 1})

# hashlib — data integrity checksums
import hashlib
def file_checksum(filepath):
    with open(filepath, "rb") as f:
        return hashlib.sha256(f.read()).hexdigest()

# argparse — CLI pipelines
import argparse
parser = argparse.ArgumentParser(description="Run ETL pipeline")
parser.add_argument("--table", required=True, help="Table to process")
parser.add_argument("--env", default="production", choices=["dev", "staging", "production"])
args = parser.parse_args()
print(f"Loading {args.table} in {args.env}")

Third-Party Packages (pip)

Installing with pip

# pip is Python's package installer — downloads from PyPI (Python Package Index)

# Install a package
# pip install pandas
# pip install requests
# pip install azure-storage-blob

# Install a specific version
# pip install pandas==2.2.0
# pip install requests>=2.28,<3.0

# Install multiple packages at once
# pip install pandas requests pyspark

# Upgrade a package
# pip install --upgrade pandas

# Uninstall a package
# pip uninstall pandas

# Show what is installed
# pip list
# pip show pandas   (detailed info about one package)

# See outdated packages
# pip list --outdated

requirements.txt

# requirements.txt — a file listing all project dependencies
# One package per line, with optional version constraints

# FILE: requirements.txt
# pandas==2.2.0
# requests>=2.28,<3.0
# pyspark==3.5.1
# azure-storage-blob==12.19.0
# python-dotenv==1.0.0
# chardet>=5.0
# openpyxl>=3.1

# Install everything from requirements.txt:
# pip install -r requirements.txt

# Generate requirements.txt from currently installed packages:
# pip freeze > requirements.txt

# WARNING: pip freeze captures EVERYTHING, including sub-dependencies
# For cleaner results, manually maintain requirements.txt with only
# your direct dependencies (what YOU installed, not their dependencies)

Pinning Versions

# Why pinning matters — a real-world disaster scenario:
# Monday: pip install pandas → installs pandas 2.1.0 → pipeline works ✅
# Wednesday: pandas 2.2.0 released with a breaking API change
# Thursday: colleague runs pip install -r requirements.txt
#           → installs pandas 2.2.0 → pipeline BREAKS ❌

# Version pinning strategies:

# EXACT pinning — safest for production
# pandas==2.2.0

# MINIMUM version — allows bug fixes
# pandas>=2.2.0

# COMPATIBLE version — allows patches, not minor bumps
# pandas~=2.2.0    (same as >=2.2.0, <2.3.0)

# RANGE — control exactly what is allowed
# pandas>=2.1.0,<2.3.0

# RECOMMENDATION for production pipelines:
# Use EXACT pinning (==) for all direct dependencies
# Run "pip install --upgrade" intentionally, test, then update the pin

Finding Packages (PyPI)

PyPI (Python Package Index) at pypi.org hosts 500,000+ packages. Key packages for data engineers: pandas (DataFrames), pyspark (Spark), requests (HTTP), boto3 (AWS), azure-storage-blob (Azure), google-cloud-storage (GCP), sqlalchemy (database ORM), pyarrow (Parquet/Arrow), python-dotenv (.env files), pytest (testing), black (formatting), and mypy (type checking).

Virtual Environments

Why Virtual Environments

Real-life analogy: Imagine you are a chef who works at two restaurants. Restaurant A uses French recipes that require butter v2.0 (a specific brand). Restaurant B uses Italian recipes that require butter v1.5 (a different brand). If you keep all your ingredients in one kitchen, you cannot have both versions of butter — one would overwrite the other. Virtual environments give each restaurant its own kitchen with its own ingredients (packages) at its own versions. They never interfere.

# WITHOUT virtual environments — global installation
# pip install pandas==2.1.0     (Project A needs 2.1.0)
# pip install pandas==2.2.0     (Project B needs 2.2.0 — OVERWRITES 2.1.0!)
# Project A is now broken

# WITH virtual environments — isolated installations
# Project A: venv_a has pandas==2.1.0
# Project B: venv_b has pandas==2.2.0
# Both work simultaneously — no conflict

Creating and Using venv

# venv is built into Python 3 — no installation needed

# Step 1: Create a virtual environment
# cd my_project
# python3 -m venv venv
# This creates a "venv" directory with its own Python and pip

# Directory structure created:
# venv/
#     bin/           (Linux/Mac) or Scripts/ (Windows)
#         python     (copy of Python interpreter)
#         pip        (isolated pip)
#         activate   (activation script)
#     lib/
#         python3.11/
#             site-packages/   (packages installed here — isolated)
#     pyvenv.cfg    (config file)

# Step 2: Activate the virtual environment
# macOS/Linux:
# source venv/bin/activate

# Windows:
# venv\Scriptsctivate

# Your prompt changes: (venv) naveen@MacBook my_project %

# Step 3: Install packages (installed ONLY in this venv)
# pip install pandas requests pyspark

# Step 4: Work normally — all imports use this venv's packages
# python pipeline.py

# Step 5: Deactivate when done
# deactivate

Activating and Deactivating

# Activating changes WHERE Python and pip look for packages:

# Before activation:
# which python → /usr/bin/python3 (system Python)
# pip list → shows GLOBAL packages

# After activation:
# which python → /home/naveen/my_project/venv/bin/python3 (venv Python)
# pip list → shows ONLY packages installed in this venv

# Deactivating reverts to the system Python:
# deactivate
# which python → /usr/bin/python3 (back to system)

# TIP: Never install packages with the system pip (without a venv active)
# You risk breaking system tools that depend on specific Python packages

Installing Packages in a Virtual Environment

# Always activate first, then install
# source venv/bin/activate
# pip install pandas==2.2.0 requests pyspark==3.5.1

# Verify installation
# pip list
# pip show pandas

# Save your dependencies
# pip freeze > requirements.txt

# Share with teammates:
# 1. They clone the repo
# 2. python3 -m venv venv
# 3. source venv/bin/activate
# 4. pip install -r requirements.txt
# 5. They have the EXACT same environment as you

# IMPORTANT: Add venv/ to .gitignore — never commit the venv directory
# .gitignore:
# venv/
# __pycache__/
# *.pyc
# .env

requirements.txt Workflow

# The complete workflow for a new project:

# 1. Create the project directory
# mkdir customer_pipeline && cd customer_pipeline

# 2. Create a virtual environment
# python3 -m venv venv

# 3. Activate it
# source venv/bin/activate

# 4. Install your direct dependencies
# pip install pandas requests azure-storage-blob python-dotenv

# 5. Save the exact versions
# pip freeze > requirements.txt

# 6. Add to .gitignore
# echo "venv/" >> .gitignore
# echo "__pycache__/" >> .gitignore

# 7. Commit requirements.txt to git (not venv/)
# git add requirements.txt .gitignore
# git commit -m "Add project dependencies"

# When a teammate joins:
# git clone repo && cd repo
# python3 -m venv venv
# source venv/bin/activate
# pip install -r requirements.txt
# → Identical environment in 30 seconds

When to Create a New Virtual Environment

• Every new project — one venv per project is the standard.
• Different Python versions — one project needs Python 3.9, another needs 3.11.
• Conflicting dependencies — Project A needs library v1.0, Project B needs v2.0.
• Production deployment — match the exact environment in CI/CD and production.
• Experimenting — try a new library without polluting your project environment.

Other Tools (conda, poetry, pipenv)

Tool	Best For	Key Difference
venv + pip	Most Python projects	Built-in, simple, standard
conda	Data science, ML	Manages Python itself + non-Python deps (C libraries). Slower but more complete
poetry	Library development	Dependency resolution, lock files, builds packages. Modern and growing
pipenv	Application development	Combines pip + virtualenv + Pipfile. Less popular now vs poetry
uv	Fast pip replacement	Rust-based, 10-100x faster than pip. Drop-in replacement

For most data engineering work, venv + pip + requirements.txt is all you need. Use conda if you work heavily with ML libraries that have complex C dependencies (TensorFlow, PyTorch). Use poetry if you are building a reusable Python package.

Data Engineering Patterns

Project Structure for a Data Pipeline

# A well-organized pipeline project:
#
# customer_pipeline/
# ├── README.md
# ├── requirements.txt
# ├── .gitignore
# ├── .env                     ← environment variables (never commit!)
# ├── config/
# │   ├── __init__.py
# │   ├── settings.py           ← configuration loading
# │   └── tables.json           ← table metadata
# ├── pipeline/
# │   ├── __init__.py
# │   ├── extract.py            ← extraction functions
# │   ├── transform.py          ← transformation functions
# │   ├── load.py               ← loading functions
# │   └── validate.py           ← validation functions
# ├── utils/
# │   ├── __init__.py
# │   ├── logging_setup.py      ← logging configuration
# │   ├── retry.py              ← retry with backoff
# │   └── db.py                 ← database connection helpers
# ├── tests/
# │   ├── __init__.py
# │   ├── test_extract.py
# │   ├── test_transform.py
# │   └── test_validate.py
# └── main.py                   ← entry point

# This structure gives you:
# - Separation of concerns (extract, transform, load in separate files)
# - Reusable utilities (retry logic, logging, DB helpers)
# - Testable components (each module can be tested independently)
# - Clean configuration (settings separate from code)
# - Standard Python packaging (importable from anywhere)

Config Module Pattern

# FILE: config/settings.py
"""Pipeline configuration — loads from environment variables and .env files."""

import os
from pathlib import Path
from dotenv import load_dotenv

# Load .env file (for local development)
env_path = Path(__file__).parent.parent / ".env"
load_dotenv(env_path)

# Database configuration
DB_HOST = os.getenv("DB_HOST", "localhost")
DB_PORT = int(os.getenv("DB_PORT", "5432"))
DB_NAME = os.getenv("DB_NAME", "warehouse")
DB_USER = os.getenv("DB_USER", "pipeline_user")
DB_PASSWORD = os.getenv("DB_PASSWORD")    # No default — must be set

# Storage configuration
STORAGE_ACCOUNT = os.getenv("STORAGE_ACCOUNT", "naveendatalake")
STORAGE_CONTAINER = os.getenv("STORAGE_CONTAINER", "bronze")

# Pipeline configuration
ENVIRONMENT = os.getenv("ENVIRONMENT", "production")
BATCH_SIZE = int(os.getenv("BATCH_SIZE", "10000"))
LOG_LEVEL = os.getenv("LOG_LEVEL", "INFO")
MAX_RETRIES = int(os.getenv("MAX_RETRIES", "3"))

# Validation
if not DB_PASSWORD:
    raise ValueError("DB_PASSWORD environment variable must be set")


# FILE: .env (local development — NEVER commit this)
# DB_HOST=server.database.windows.net
# DB_PORT=1433
# DB_NAME=warehouse
# DB_USER=pipeline_admin
# DB_PASSWORD=secret123
# ENVIRONMENT=development
# LOG_LEVEL=DEBUG


# Usage in pipeline code:
from config.settings import DB_HOST, DB_NAME, BATCH_SIZE, LOG_LEVEL

Utils Module Pattern

# FILE: utils/retry.py
"""Retry logic with exponential backoff — reusable across all pipelines."""

import time
import logging

logger = logging.getLogger(__name__)

def retry(func, max_attempts=3, base_delay=1, exceptions=(Exception,)):
    """Execute a function with retry and exponential backoff."""
    for attempt in range(1, max_attempts + 1):
        try:
            return func()
        except exceptions as e:
            if attempt == max_attempts:
                logger.error(f"All {max_attempts} attempts failed: {e}")
                raise
            delay = base_delay * (2 ** (attempt - 1))
            logger.warning(f"Attempt {attempt} failed: {e}. Retrying in {delay}s...")
            time.sleep(delay)


# FILE: utils/db.py
"""Database connection helpers."""

def get_connection_string():
    from config.settings import DB_HOST, DB_PORT, DB_NAME, DB_USER, DB_PASSWORD
    return f"host={DB_HOST} port={DB_PORT} dbname={DB_NAME} user={DB_USER} password={DB_PASSWORD}"


# FILE: utils/__init__.py — re-export commonly used utilities
from utils.retry import retry
from utils.db import get_connection_string

# Now pipeline code can do:
# from utils import retry, get_connection_string

Pipeline Package with Extract/Transform/Load Modules

# FILE: pipeline/extract.py
import logging
from utils import retry, get_connection_string

logger = logging.getLogger(__name__)

def extract_table(table_name):
    """Extract a full table from the source database."""
    conn_str = get_connection_string()
    logger.info(f"Extracting: {table_name}")

    def _do_extract():
        conn = connect(conn_str)
        return conn.execute(f"SELECT * FROM {table_name}").fetchall()

    # Retry on connection failures
    data = retry(_do_extract, max_attempts=3)
    logger.info(f"Extracted {len(data):,} rows from {table_name}")
    return data


# FILE: pipeline/transform.py
import logging
from datetime import datetime

logger = logging.getLogger(__name__)

def apply_standard_transforms(data, table_name):
    """Apply standard transformations to any table."""
    logger.info(f"Transforming {len(data):,} rows for {table_name}")

    for row in data:
        row["_loaded_at"] = datetime.now().isoformat()
        row["_source_table"] = table_name
        # Strip whitespace from string fields
        for key, value in row.items():
            if isinstance(value, str):
                row[key] = value.strip()

    logger.info(f"Transformation complete: {len(data):,} rows")
    return data


# FILE: main.py — the entry point that ties everything together
import logging
from config.settings import LOG_LEVEL, ENVIRONMENT
from utils.logging_setup import configure_logging
from pipeline.extract import extract_table
from pipeline.transform import apply_standard_transforms
from pipeline.load import load_to_warehouse

logger = configure_logging("customer_pipeline", level=LOG_LEVEL)

def main():
    logger.info(f"Pipeline starting (env={ENVIRONMENT})")

    tables = ["customers", "orders", "products"]
    for table in tables:
        data = extract_table(table)
        data = apply_standard_transforms(data, table)
        load_to_warehouse(data, f"silver.{table}")

    logger.info("Pipeline complete")

if __name__ == "__main__":
    main()

Shared Library Across Multiple Projects

# When multiple pipelines share the same utilities (retry, logging, DB helpers),
# create a shared library that can be installed in each project's venv:

# shared_utils/
# ├── setup.py               ← makes it pip-installable
# ├── shared_utils/
# │   ├── __init__.py
# │   ├── retry.py
# │   ├── logging_setup.py
# │   └── db.py

# FILE: setup.py
from setuptools import setup, find_packages

setup(
    name="shared-utils",
    version="1.0.0",
    packages=find_packages(),
    install_requires=[
        "python-dotenv>=1.0",
    ],
    python_requires=">=3.9",
)

# Install in any project's venv:
# pip install -e /path/to/shared_utils
# or from a private git repo:
# pip install git+https://github.com/naveen/shared-utils.git

# Now use it in any project:
# from shared_utils.retry import retry
# from shared_utils.logging_setup import configure_logging

Common Mistakes

1. Naming files after standard library modules — creating json.py, csv.py, or logging.py shadows the standard library. Your import json imports your file instead of Python’s json module. Name your files descriptively: json_utils.py, csv_processor.py.
2. Not using virtual environments — installing everything globally causes version conflicts between projects. Always create a venv: python3 -m venv venv. It takes 5 seconds and prevents hours of debugging.
3. Committing the venv directory to Git — the venv directory contains thousands of files specific to your OS and Python version. It does not belong in version control. Add venv/ to .gitignore and commit requirements.txt instead.
4. Using from module import * — star imports pollute the namespace, cause name collisions, and make code unreadable. Import only what you need: from utils import clean_string, parse_date.
5. Not using if __name__ == "__main__" — without the guard, test code in your modules runs every time another file imports them. Always wrap executable code (tests, CLI, demo) in the guard.
6. Circular imports — module A imports module B, and module B imports module A. Fix by restructuring (move shared code to a third module), using lazy imports (import inside functions), or combining tightly coupled modules.
7. Not pinning dependency versions — pip install pandas installs the latest version. Next month, a breaking change in pandas breaks your pipeline. Always pin: pandas==2.2.0 in requirements.txt.

Interview Questions

Q: What is the difference between a module and a package? A: A module is a single .py file that contains functions, classes, and variables. A package is a directory of modules with an __init__.py file. Packages let you organize related modules into a hierarchy — for example, pipeline/extract.py and pipeline/transform.py inside a pipeline package. You import from packages using dot notation: from pipeline.extract import extract_from_sql.

Q: What is __init__.py and is it required? A: __init__.py marks a directory as a Python package. It runs when the package is first imported. It can be empty (just marks the directory) or can re-export commonly used items for a cleaner API. In Python 3.3+, it is technically optional (namespace packages), but including it is still best practice because it makes the package structure explicit and enables IDE support.

Q: What does if __name__ == "__main__" do? A: It checks whether the file is being run directly (python script.py) or imported by another module. When run directly, __name__ equals "__main__" and the guarded code executes. When imported, __name__ equals the module name and the guarded code is skipped. This prevents test code and CLI logic from running when the module is imported as a library.

Q: What is a virtual environment and why do you need one? A: A virtual environment is an isolated Python installation with its own packages and versions. It prevents dependency conflicts between projects — Project A can use pandas 2.1.0 while Project B uses pandas 2.2.0. Create one with python3 -m venv venv, activate with source venv/bin/activate, and install packages with pip install. The venv directory should be in .gitignore; share requirements.txt instead.

Q: What is the difference between absolute and relative imports? A: Absolute imports use the full path from the project root: from pipeline.extract import extract_from_sql. Relative imports use dots to refer to the current package: from .extract import extract_from_sql (one dot = current package). PEP 8 recommends absolute imports because they are unambiguous and work in all contexts. Relative imports only work inside packages.

Q: How do you handle circular imports? A: Three strategies: (1) Move the shared code to a third module that both can import. (2) Use lazy imports — import inside the function that needs it, not at the module level. (3) Combine tightly coupled modules into one. Circular imports are usually a sign of poor code organization — fix the structure rather than patching the import.

Q: What is requirements.txt and how do you use it? A: requirements.txt lists all project dependencies with version pins: pandas==2.2.0. Generate it with pip freeze > requirements.txt. Teammates recreate the exact environment with pip install -r requirements.txt. Always pin exact versions for production to prevent breaking changes from new releases.

Wrapping Up

Modules, packages, and virtual environments are the organizational backbone of professional Python development. Modules split your code into reusable files. Packages organize modules into directories. __init__.py defines the package’s public API. Virtual environments isolate dependencies per project. And requirements.txt makes your environment reproducible.

For data engineering, the key patterns are: one module per concern (extract, transform, load, validate), a config module for environment-specific settings loaded from .env files, a utils package for shared logic (retry, logging, database helpers), and a clean main.py entry point that ties everything together. Pin your dependencies, never commit your venv, and always use the __name__ == "__main__" guard.

Previous in this series: OOP Advanced — Inheritance, Polymorphism, Abstract Classes

Next in this series: Dates, Times, Timezones — datetime, timedelta, and Scheduling

---

Naveen Vuppula is a Senior Data Engineering Consultant and app developer based in Ontario, Canada. He writes about Python, SQL, AWS, Azure, and everything data engineering at DriveDataScience.com.