Python Functions: Parameters, Return Values, Scope, Default Arguments, *args, **kwargs, First-Class Functions, Closures, and Docstrings
Functions are how you organize code into reusable, testable, named blocks. Without functions, a data pipeline would be one giant script — 500 lines of reading, cleaning, transforming, and loading, all tangled together. With functions, each step is isolated: read_source(), clean_data(), transform(), load_to_delta(). Each function does one thing, can be tested independently, and can be reused across pipelines.
But Python functions have depth beyond basic def and return. Default arguments have a mutable trap that catches everyone. *args and **kwargs enable flexible APIs. Scope rules (LEGB) determine which variable wins when names collide. And functions themselves are objects — you can pass them as arguments, return them from other functions, and store them in lists.
Think of a function like a vending machine. You insert inputs (arguments), the machine does internal work (function body), and it dispenses an output (return value). You do not need to know HOW the machine works internally — you just need to know what to put in and what comes out. That separation of “interface” from “implementation” is the core idea behind functions.
Table of Contents
- Defining and Calling Functions
- Basic Function
- Functions with Parameters
- Return Values
- Returning Multiple Values
- Returning None (Implicit vs Explicit)
- Parameters and Arguments
- Positional vs Keyword Arguments
- Default Parameter Values
- The Mutable Default Argument Trap
- Keyword-Only Arguments
- Positional-Only Arguments (Python 3.8+)
- *args — Variable Positional Arguments
- How *args Works
- Practical Uses of *args
- **kwargs — Variable Keyword Arguments
- How **kwargs Works
- Combining *args and **kwargs
- The Full Parameter Order
- Scope and the LEGB Rule
- Local Scope
- Enclosing Scope (Nested Functions)
- Global Scope
- Built-in Scope
- The global Keyword
- The nonlocal Keyword
- Why Global Variables Are Dangerous
- Functions Are First-Class Objects
- Assigning Functions to Variables
- Passing Functions as Arguments
- Returning Functions from Functions
- Storing Functions in Data Structures
- Lambda Functions (Preview)
- Closures
- What Is a Closure
- Practical Closure Patterns
- Docstrings and Type Hints
- Writing Good Docstrings
- Type Hints (Annotations)
- Recursive Functions
- How Recursion Works
- Recursion vs Iteration
- Functions in Data Engineering
- Pipeline Building Blocks
- Validation Functions
- Retry Wrapper
- Common Mistakes
- Interview Questions
- Wrapping Up
Defining and Calling Functions
Basic Function
# def keyword + function name + parentheses + colon
def greet():
print("Hello, World!")
# Calling the function — name + parentheses
greet() # "Hello, World!"
greet() # "Hello, World!" — can call as many times as you want
# The function is an OBJECT — it exists even before you call it
print(type(greet)) # <class 'function'>
print(greet) # <function greet at 0x7f8b...> — has a memory addressReal-life analogy: def greet(): is like writing a recipe card. The recipe exists on the card but no food is prepared. greet() is like actually cooking the recipe — only then does something happen. You can cook the same recipe as many times as you want.
Functions with Parameters
# Parameters are variables listed in the function definition
# Arguments are the actual values passed when calling
def greet(name): # "name" is a PARAMETER
print(f"Hello, {name}!")
greet("Naveen") # "Naveen" is an ARGUMENT
greet("Alice") # Different argument, same function
# Multiple parameters
def add(a, b):
result = a + b
print(f"{a} + {b} = {result}")
add(10, 20) # 10 + 20 = 30
add(3, 7) # 3 + 7 = 10Return Values
# return sends a value BACK to the caller
def add(a, b):
return a + b # Sends the result back — does NOT print it
result = add(10, 20)
print(result) # 30
# return immediately EXITS the function — code after return never runs
def check_age(age):
if age < 0:
return "Invalid" # Exits here if age is negative
if age >= 18:
return "Adult" # Exits here if adult
return "Minor" # Only reaches here if 0-17
print(check_age(25)) # "Adult"
print(check_age(-5)) # "Invalid"
print(check_age(10)) # "Minor"Returning Multiple Values
# Python can return multiple values as a TUPLE
def get_stats(numbers):
return min(numbers), max(numbers), sum(numbers) / len(numbers)
low, high, avg = get_stats([10, 20, 30, 40, 50])
print(f"Min: {low}, Max: {high}, Avg: {avg}")
# Min: 10, Max: 50, Avg: 30.0
# Under the hood, it returns a tuple:
result = get_stats([10, 20, 30])
print(type(result)) # <class 'tuple'>
print(result) # (10, 30, 20.0)
# Practical: return status and data together
def load_data(path):
try:
data = read_file(path)
return True, data, None
except Exception as e:
return False, None, str(e)
success, data, error = load_data("/data/sales.csv")
if success:
process(data)
else:
print(f"Failed: {error}")Returning None (Implicit vs Explicit)
# If a function has no return statement, it returns None
def greet(name):
print(f"Hello, {name}!")
# No return statement
result = greet("Naveen")
print(result) # None
# Explicit None return — signals "no result" intentionally
def find_user(user_id):
for user in users:
if user["id"] == user_id:
return user
return None # Explicit: "not found"
# Functions that modify data in place (like list.sort()) return None
# This is a Python CONVENTION — mutating functions return None to signal
# "I modified the original, I did not create a new thing"
my_list = [3, 1, 2]
result = my_list.sort()
print(result) # None — sort() returns None
print(my_list) # [1, 2, 3] — but the list is sortedParameters and Arguments
Positional vs Keyword Arguments
def create_user(name, age, city):
return {"name": name, "age": age, "city": city}
# POSITIONAL arguments — matched by position (order matters)
user = create_user("Naveen", 30, "Toronto")
# KEYWORD arguments — matched by name (order does NOT matter)
user = create_user(city="Toronto", name="Naveen", age=30) # Same result!
# MIX — positional first, then keyword
user = create_user("Naveen", city="Toronto", age=30) # Valid
# user = create_user(name="Naveen", 30, "Toronto") # SyntaxError!
# Rule: positional args MUST come before keyword argsReal-life analogy: Positional arguments are like a drive-through order — “I will have a burger, fries, and a coke” (the cashier knows: first item = main, second = side, third = drink). Keyword arguments are like ordering at a sit-down restaurant — “I would like the coke for my drink, the burger as my main, and fries as my side” (order does not matter because you named each item).
Default Parameter Values
# Parameters can have defaults — used when the caller does not provide a value
def create_user(name, age, city="Toronto", active=True):
return {"name": name, "age": age, "city": city, "active": active}
# Defaults used
user = create_user("Naveen", 30)
print(user) # {'name': 'Naveen', 'age': 30, 'city': 'Toronto', 'active': True}
# Override one default
user = create_user("Alice", 25, city="London")
print(user) # {'name': 'Alice', 'age': 25, 'city': 'London', 'active': True}
# Override all defaults
user = create_user("Bob", 35, "Delhi", False)
print(user) # {'name': 'Bob', 'age': 35, 'city': 'Delhi', 'active': False}
# Rule: parameters WITH defaults must come AFTER parameters WITHOUT defaults
# def bad(city="Toronto", name): # SyntaxError!The Mutable Default Argument Trap
This is one of Python’s most infamous traps. It catches experienced developers, causes subtle bugs, and is asked in every interview.
# ❌ DANGEROUS — mutable default argument
def add_item(item, items=[]): # Default [] is created ONCE at function definition
items.append(item)
return items
print(add_item("apple")) # ['apple'] — looks correct
print(add_item("banana")) # ['apple', 'banana'] — WAIT, where did apple come from?!
print(add_item("cherry")) # ['apple', 'banana', 'cherry'] — accumulating across calls!
# WHY? The default list [] is created ONCE when Python defines the function.
# Every call that uses the default SHARES the same list object.
# It is NOT re-created on each call.# ✅ CORRECT — use None as default, create list inside the function
def add_item(item, items=None):
if items is None:
items = [] # NEW list created on EACH call
items.append(item)
return items
print(add_item("apple")) # ['apple']
print(add_item("banana")) # ['banana'] — fresh list each time!
print(add_item("cherry")) # ['cherry'] — no accumulation
# RULE: Never use mutable objects (list, dict, set) as default arguments.
# Always use None and create the mutable object inside the function.Real-life analogy: The mutable default is like a shared notepad by the office printer. Everyone who uses the printer writes on the same notepad. The first person writes “apple.” The second person sees “apple” and adds “banana.” Nobody gets a fresh notepad — they all share one. The fix is: give each person their own notepad (create a new list inside the function).
Keyword-Only Arguments
# Arguments AFTER * must be passed as keywords — prevents positional mistakes
def connect(host, port, *, timeout=30, retry=True, ssl=True):
print(f"Connecting to {host}:{port} (timeout={timeout}, ssl={ssl})")
# Valid:
connect("server.com", 1433, timeout=60, ssl=False)
# Invalid:
# connect("server.com", 1433, 60, True, False) # TypeError!
# After *, Python forces you to NAME the arguments — prevents confusion
# about which positional arg is timeout vs retry vs ssl
# Why this matters: without keyword-only, connect("server.com", 1433, 60, True, False)
# is ambiguous — is 60 the timeout, retry count, or something else?Positional-Only Arguments (Python 3.8+)
# Arguments BEFORE / must be passed positionally — cannot use keyword
def power(base, exp, /):
return base ** exp
# Valid:
print(power(2, 10)) # 1024
# Invalid:
# print(power(base=2, exp=10)) # TypeError! Cannot use keyword
# Why? Some functions have parameter names that are implementation details.
# Built-in functions like len() use this — you cannot write len(obj=[1,2,3])
# Complete signature with all zones:
def func(pos_only, /, normal, *, kw_only):
pass
# pos_only: must be positional
# normal: can be either positional or keyword
# kw_only: must be keyword*args — Variable Positional Arguments
How *args Works
# *args collects EXTRA positional arguments into a TUPLE
def add_all(*args):
print(f"Type: {type(args)}") # <class 'tuple'>
print(f"Values: {args}")
return sum(args)
print(add_all(1, 2, 3)) # Type: tuple, Values: (1, 2, 3), Returns: 6
print(add_all(10, 20, 30, 40)) # Type: tuple, Values: (10, 20, 30, 40), Returns: 100
print(add_all(5)) # Type: tuple, Values: (5,), Returns: 5
print(add_all()) # Type: tuple, Values: (), Returns: 0
# "args" is just a convention — any name works after *
def my_func(*numbers): # *numbers works the same as *args
return sum(numbers)
# Mix with regular parameters — *args captures the REMAINING positional args
def greet(greeting, *names):
for name in names:
print(f"{greeting}, {name}!")
greet("Hello", "Alice", "Bob", "Charlie")
# Hello, Alice!
# Hello, Bob!
# Hello, Charlie!Practical Uses of *args
# Wrapper function that passes args through
def log_and_call(func, *args):
print(f"Calling {func.__name__} with args: {args}")
return func(*args) # Unpack args back into individual arguments
def add(a, b):
return a + b
result = log_and_call(add, 10, 20)
# "Calling add with args: (10, 20)"
print(result) # 30
# Flexible print function
def debug_print(*values, prefix="DEBUG"):
formatted = " | ".join(str(v) for v in values)
print(f"[{prefix}] {formatted}")
debug_print("loading data", 42, True)
# [DEBUG] loading data | 42 | True**kwargs — Variable Keyword Arguments
How **kwargs Works
# **kwargs collects EXTRA keyword arguments into a DICT
def create_record(**kwargs):
print(f"Type: {type(kwargs)}") # <class 'dict'>
print(f"Values: {kwargs}")
return kwargs
record = create_record(name="Naveen", age=30, city="Toronto")
# Type: dict
# Values: {'name': 'Naveen', 'age': 30, 'city': 'Toronto'}
# Practical: flexible configuration
def connect_db(**config):
host = config.get("host", "localhost")
port = config.get("port", 5432)
db = config.get("database", "default")
print(f"Connecting to {host}:{port}/{db}")
connect_db(host="server.com", port=1433, database="sales")
connect_db(host="backup.com") # Uses defaults for port and databaseCombining *args and **kwargs
# Accept ANY combination of arguments — the most flexible signature
def super_flexible(*args, **kwargs):
print(f"Positional: {args}")
print(f"Keyword: {kwargs}")
super_flexible(1, 2, 3, name="Naveen", active=True)
# Positional: (1, 2, 3)
# Keyword: {'name': 'Naveen', 'active': True}
# Practical: wrapper that forwards ALL arguments to another function
def retry(func, max_attempts=3, *args, **kwargs):
for attempt in range(1, max_attempts + 1):
try:
return func(*args, **kwargs)
except Exception as e:
print(f"Attempt {attempt} failed: {e}")
if attempt == max_attempts:
raiseThe Full Parameter Order
# When combining all parameter types, the order is FIXED:
def func(pos_only, /, normal, *args, kw_only, **kwargs):
pass
# Order:
# 1. Positional-only (before /)
# 2. Normal (positional or keyword)
# 3. *args (variable positional)
# 4. Keyword-only (after * or *args)
# 5. **kwargs (variable keyword — always LAST)
# Most common in practice:
def func(required, optional="default", *args, **kwargs):
passScope and the LEGB Rule
Local Scope
# Variables created INSIDE a function exist ONLY inside that function
def my_function():
x = 10 # Local variable — born and dies inside this function
print(f"Inside: {x}")
my_function() # "Inside: 10"
# print(x) # NameError: name 'x' is not defined — x does not exist here
# Parameters are also local variables
def add(a, b): # a and b are local to add()
result = a + b # result is local to add()
return result
add(10, 20)
# print(a) # NameError — a does not exist outside add()Enclosing Scope (Nested Functions)
# A nested function can READ variables from the enclosing function
def outer():
message = "Hello from outer" # Enclosing scope variable
def inner():
print(message) # Reads from enclosing scope — works!
inner()
outer() # "Hello from outer"
# The inner function CANNOT modify the enclosing variable (without nonlocal)
def outer():
count = 0
def inner():
# count += 1 # UnboundLocalError! Python thinks count is local
print(count) # Reading is fine
inner()Global Scope
# Variables defined at module level (outside any function) are global
DB_HOST = "server.database.windows.net" # Global variable
def connect():
print(f"Connecting to {DB_HOST}") # Can READ global variables
# DB_HOST = "other.com" # This creates a LOCAL variable, does NOT modify global!
connect() # "Connecting to server.database.windows.net"Built-in Scope
# Python's built-in names: print, len, type, range, int, str, list, etc.
# These are available everywhere — the outermost scope
# If you accidentally shadow a built-in:
list = [1, 2, 3] # Now "list" refers to YOUR variable, not the built-in!
# new_list = list("hello") # TypeError: 'list' object is not callable
# Fix: del list (restores the built-in)
# LEGB lookup order:
# 1. Local — inside the current function
# 2. Enclosing — inside any enclosing functions
# 3. Global — module-level variables
# 4. Built-in — Python built-in names (print, len, type, etc.)
# Python searches in THIS order. The FIRST match wins.LEGB Rule Visualized:
┌─────────────────────────────────────────────┐
│ BUILT-IN: print, len, type, range, int, ... │
│ ┌──────────────────────────────────────┐ │
│ │ GLOBAL: DB_HOST, config, constants │ │
│ │ ┌──────────────────────────────┐ │ │
│ │ │ ENCLOSING: outer function │ │ │
│ │ │ ┌──────────────────────┐ │ │ │
│ │ │ │ LOCAL: inner function│ │ │ │
│ │ │ │ x, y, result │ │ │ │
│ │ │ └──────────────────────┘ │ │ │
│ │ └──────────────────────────────┘ │ │
│ └──────────────────────────────────────┘ │
└─────────────────────────────────────────────┘
When Python sees a variable name, it searches:
Local → Enclosing → Global → Built-in (first match wins)The global Keyword
# global lets a function MODIFY a global variable
counter = 0
def increment():
global counter # "I want to modify the GLOBAL counter, not create a local one"
counter += 1
increment()
increment()
increment()
print(counter) # 3
# WITHOUT global, counter += 1 would create a LOCAL variable and crash:
# UnboundLocalError: local variable 'counter' referenced before assignmentThe nonlocal Keyword
# nonlocal lets a nested function MODIFY an enclosing function's variable
def make_counter():
count = 0
def increment():
nonlocal count # "I want to modify the ENCLOSING count"
count += 1
return count
return increment
counter = make_counter()
print(counter()) # 1
print(counter()) # 2
print(counter()) # 3
# Each call modifies the same "count" variable in the enclosing scopeWhy Global Variables Are Dangerous
# Globals make code unpredictable — any function can change them at any time
total = 0
def add_to_total(amount):
global total
total += amount
def reset_total():
global total
total = 0
add_to_total(50)
add_to_total(30)
reset_total() # Oops — someone else called reset!
add_to_total(20)
print(total) # 20, not 100! reset_total silently wiped the data
# RULE: Avoid global variables. Pass data INTO functions as parameters
# and get data OUT via return values. Functions should be self-contained.
# ✅ GOOD — pure function, no global state
def calculate_total(amounts):
return sum(amounts)
# ❌ BAD — relies on global state
# total = 0
# def add(amount): global total; total += amountFunctions Are First-Class Objects
In Python, functions are objects — they have a type, a memory address, and can be assigned to variables, passed as arguments, returned from other functions, and stored in data structures. This is called “first-class functions” and it enables powerful patterns.
Assigning Functions to Variables
def shout(text):
return text.upper()
# Assign function to a variable (no parentheses — the function itself, not its result)
yell = shout # yell is now another name for the same function
print(yell("hello")) # "HELLO"
print(yell is shout) # True — same function object
# With parentheses: shout("hello") CALLS the function and returns the result
# Without parentheses: shout IS the function object itselfPassing Functions as Arguments
# Functions can be passed as arguments to other functions
def apply_to_list(func, data):
return [func(item) for item in data]
names = ["alice", "bob", "charlie"]
print(apply_to_list(str.upper, names)) # ['ALICE', 'BOB', 'CHARLIE']
print(apply_to_list(len, names)) # [5, 3, 7]
# This is how sorted(key=...) works:
students = [("Alice", 88), ("Bob", 72), ("Charlie", 95)]
sorted_students = sorted(students, key=lambda s: s[1])
# key receives a FUNCTION — sorted calls it on each item to determine order
# Practical: apply different cleaning functions
def trim(s): return s.strip()
def lower(s): return s.lower()
def remove_special(s): return ''.join(c for c in s if c.isalnum() or c.isspace())
cleaners = [trim, lower, remove_special]
text = " Hello, World! @#$ "
for cleaner in cleaners:
text = cleaner(text)
print(text) # "hello world "Returning Functions from Functions
# A function can CREATE and RETURN a new function
def make_multiplier(factor):
def multiply(x):
return x * factor
return multiply # Return the function object (not the result)
double = make_multiplier(2)
triple = make_multiplier(3)
print(double(5)) # 10
print(triple(5)) # 15
print(double(100)) # 200
# double and triple are DIFFERENT functions, each "remembering" its factor
# This is a CLOSURE — covered in the next sectionStoring Functions in Data Structures
# Functions can be stored in lists, dicts, etc.
def add(a, b): return a + b
def sub(a, b): return a - b
def mul(a, b): return a * b
def div(a, b): return a / b if b != 0 else None
# Calculator using a dict of functions
operations = {"+": add, "-": sub, "*": mul, "/": div}
operator = "+"
result = operations[operator](10, 3)
print(result) # 13
# Practical: dispatch table for pipeline operations
pipeline_steps = {
"clean": clean_data,
"validate": validate_data,
"transform": transform_data,
"load": load_to_delta,
}
for step_name, step_func in pipeline_steps.items():
print(f"Running: {step_name}")
data = step_func(data)Lambda Functions (Preview)
# Lambda: one-line anonymous functions
# Syntax: lambda parameters: expression
square = lambda x: x ** 2
print(square(5)) # 25
# Same as:
def square(x):
return x ** 2
# Used mostly inline — as arguments to sorted(), map(), filter()
names = ["Charlie", "Alice", "Bob"]
sorted_names = sorted(names, key=lambda n: len(n))
print(sorted_names) # ['Bob', 'Alice', 'Charlie']
# Lambda covered in depth in the next post (Post 10)Closures
What Is a Closure
A closure is a function that remembers the variables from the enclosing scope even after the outer function has finished executing. The inner function “closes over” the enclosing variables — hence the name.
def make_greeter(greeting):
# greeting is in the enclosing scope
def greet(name):
return f"{greeting}, {name}!" # "remembers" greeting
return greet
hello_greeter = make_greeter("Hello")
hi_greeter = make_greeter("Hi")
print(hello_greeter("Naveen")) # "Hello, Naveen!"
print(hi_greeter("Alice")) # "Hi, Alice!"
# make_greeter has already RETURNED — its local scope is "gone"
# But hello_greeter still remembers greeting="Hello"
# That is a closure — the inner function captures the enclosing variableReal-life analogy: A closure is like a letter with a pre-printed letterhead. The office (outer function) prints the letterhead (“Hello, “) and hands you blank letters (inner function). Even after you leave the office, each letter still has the letterhead. The letterhead is the “closed-over” variable.
Practical Closure Patterns
# Pattern 1: Configuration factory
def make_logger(level):
def log(message):
print(f"[{level}] {message}")
return log
info = make_logger("INFO")
error = make_logger("ERROR")
info("Pipeline started") # [INFO] Pipeline started
error("Connection failed") # [ERROR] Connection failed
# Pattern 2: Counter
def make_counter(start=0):
count = [start] # Using list to allow mutation (closure captures the list)
def increment():
count[0] += 1
return count[0]
return increment
counter = make_counter()
print(counter()) # 1
print(counter()) # 2
# Pattern 3: Data validation with configurable rules
def make_validator(min_val, max_val):
def validate(value):
return min_val <= value <= max_val
return validate
age_validator = make_validator(0, 150)
salary_validator = make_validator(0, 10_000_000)
print(age_validator(25)) # True
print(age_validator(-5)) # False
print(salary_validator(75000)) # TrueDocstrings and Type Hints
Writing Good Docstrings
# Docstrings document what a function does, its parameters, and return value
def clean_customer_data(df, fill_nulls=True, dedup_key="customer_id"):
"""
Clean raw customer data for the Silver layer.
Applies standardization (trim, lowercase email), handles nulls,
and removes duplicates keeping the most recent record.
Args:
df: Raw PySpark DataFrame from Bronze layer.
fill_nulls: If True, fill null city/country with 'Unknown'. Default True.
dedup_key: Column name to deduplicate on. Default 'customer_id'.
Returns:
Cleaned PySpark DataFrame ready for Silver layer.
Raises:
ValueError: If dedup_key column does not exist in the DataFrame.
Example:
clean_df = clean_customer_data(raw_df, fill_nulls=True)
"""
pass
# Access the docstring
print(clean_customer_data.__doc__)
help(clean_customer_data) # Pretty-printed documentationType Hints (Annotations)
# Type hints document expected types — NOT enforced at runtime, but help editors and linters
def add(a: int, b: int) -> int:
return a + b
def greet(name: str, times: int = 1) -> str:
return (f"Hello, {name}! " * times).strip()
def find_user(user_id: int) -> dict | None: # Python 3.10+
"""Returns user dict or None if not found."""
pass
# With complex types
from typing import Optional
def process_data(
records: list[dict],
batch_size: int = 100,
callback: Optional[callable] = None
) -> tuple[int, int]:
"""Returns (success_count, error_count)."""
pass
# Type hints are documentation — Python does NOT enforce them
add("hello", "world") # Works! Python allows it, but mypy/pylint would flag itRecursive Functions
How Recursion Works
# A recursive function calls ITSELF to solve smaller subproblems
def factorial(n):
if n <= 1: # BASE CASE — stops the recursion
return 1
return n * factorial(n - 1) # RECURSIVE CASE — calls itself
print(factorial(5)) # 120 (5 × 4 × 3 × 2 × 1)
# How it unfolds:
# factorial(5) → 5 × factorial(4)
# factorial(4) → 4 × factorial(3)
# factorial(3) → 3 × factorial(2)
# factorial(2) → 2 × factorial(1)
# factorial(1) → 1 (base case — stops here)
# Then it "unwinds": 2×1 = 2, 3×2 = 6, 4×6 = 24, 5×24 = 120
# Practical: flatten a nested dict
def flatten_dict(d, parent_key='', sep='.'):
items = []
for k, v in d.items():
new_key = f"{parent_key}{sep}{k}" if parent_key else k
if isinstance(v, dict):
items.extend(flatten_dict(v, new_key, sep).items()) # Recursive call
else:
items.append((new_key, v))
return dict(items)Real-life analogy: Recursion is like Russian nesting dolls (matryoshka). You open the outer doll (first call), find a smaller doll inside (recursive call), open that, find an even smaller one, until you reach the tiny solid doll (base case). Then you put them back together (return values unwind).
Recursion vs Iteration
| Feature | Recursion | Iteration (Loops) |
|---|---|---|
| Readability | Elegant for tree/nested structures | Clearer for linear sequences |
| Performance | Overhead per call (stack frames) | Generally faster |
| Memory | Uses call stack (limited to ~1000 depth) | Constant memory |
| Risk | RecursionError if too deep | Infinite loop if condition never met |
| Best for | Trees, nested dicts, divide-and-conquer | Lists, ranges, sequential processing |
# Python default recursion limit: 1000 calls deep
import sys
print(sys.getrecursionlimit()) # 1000
# RULE: Prefer iteration for simple sequences. Use recursion for hierarchical/nested data.Functions in Data Engineering
Pipeline Building Blocks
# Each pipeline step is a function — clean, testable, reusable
def read_bronze(table_name):
"""Read raw data from Bronze layer."""
return spark.table(f"bronze.{table_name}")
def clean_data(df, fill_nulls=True):
"""Apply standard cleaning transformations."""
df = df.dropDuplicates()
if fill_nulls:
df = df.fillna({"city": "Unknown", "country": "Unknown"})
return df
def write_silver(df, table_name):
"""Write cleaned data to Silver layer."""
df.write.format("delta").mode("overwrite").saveAsTable(f"silver.{table_name}")
return df.count()
# Pipeline = function composition
def run_pipeline(table_name):
raw = read_bronze(table_name)
cleaned = clean_data(raw)
rows = write_silver(cleaned, table_name)
print(f"Pipeline complete: {rows} rows written to silver.{table_name}")
run_pipeline("customers")Validation Functions
def validate_record(record, required_fields=None):
"""Validate a data record against required fields and business rules."""
if required_fields is None:
required_fields = ["customer_id", "email"]
errors = []
for field in required_fields:
if not record.get(field):
errors.append(f"Missing required field: {field}")
if record.get("email") and "@" not in record["email"]:
errors.append(f"Invalid email: {record['email']}")
if record.get("age") is not None and not (0 <= record["age"] <= 150):
errors.append(f"Invalid age: {record['age']}")
return {"valid": len(errors) == 0, "errors": errors}
# Usage
print(validate_record({"customer_id": 1, "email": "test"}))
# {'valid': False, 'errors': ['Invalid email: test']}Retry Wrapper
import time
def with_retry(func, max_retries=3, delay=1, *args, **kwargs):
"""Call a function with automatic retry on failure."""
for attempt in range(1, max_retries + 1):
try:
return func(*args, **kwargs)
except Exception as e:
print(f"Attempt {attempt}/{max_retries} failed: {e}")
if attempt == max_retries:
raise
time.sleep(delay * attempt) # Linear backoff
# Usage
result = with_retry(fetch_data, max_retries=5, delay=2, url="https://api.example.com")
# Automatically retries up to 5 times with increasing delaysCommon Mistakes
- Mutable default arguments —
def f(items=[])shares ONE list across all calls. Usedef f(items=None): items = items or []. This is the most common function-related bug in Python. - Forgetting to return a value — a function without
returnreturnsNone.result = my_function()isNoneif you forgot the return statement. Always check. - Confusing
print()withreturn—print()displays output to the screen.returnsends a value back to the caller. A function that prints but does not return givesNonewhen assigned. - Modifying a global variable without
global— Python creates a LOCAL variable instead, leading to UnboundLocalError. If you must modify a global (avoid if possible), use theglobalkeyword. - Shadowing built-in names —
list = [1, 2, 3]overwrites the built-inlistfunction. Avoid naming variableslist,dict,str,type,id,input,print. - Using
returninside a loop when you wantbreak—returnexits the ENTIRE function.breakexits only the loop. Make sure you know which one you intend. - Not writing docstrings — undocumented functions become mysteries within weeks. Always write at least a one-line docstring explaining what the function does.
Interview Questions
Q: What is the mutable default argument trap?
A: Default mutable objects (lists, dicts, sets) are created ONCE at function definition time and shared across all calls. def f(items=[]) accumulates items across calls because every call that uses the default shares the same list. Fix: use None as default and create the mutable object inside: def f(items=None): items = items or [].
Q: What is the difference between *args and **kwargs?
A: *args collects extra positional arguments into a tuple. **kwargs collects extra keyword arguments into a dict. Together they allow a function to accept any combination of arguments. Use *args for variable-length positional data and **kwargs for optional named configuration.
Q: Explain the LEGB scope rule. A: When Python encounters a variable name, it searches four scopes in order: Local (inside the current function), Enclosing (inside any outer functions), Global (module level), Built-in (Python built-ins like print and len). The first match wins. This determines which variable a name refers to when the same name exists at multiple levels.
Q: What is a closure in Python? A: A closure is an inner function that remembers variables from the enclosing function’s scope even after the outer function has returned. The inner function “closes over” the enclosing variables. Used for factory functions, configuration-based function generation, and maintaining state without classes.
Q: What are first-class functions?
A: In Python, functions are objects — they have a type (function), can be assigned to variables, passed as arguments to other functions, returned from functions, and stored in data structures (lists, dicts). This enables patterns like callbacks, factory functions, dispatch tables, and higher-order functions like sorted(key=func).
Q: Why should you avoid global variables? A: Globals create hidden dependencies — any function can read or modify them, making bugs hard to track. A function that uses globals is not self-contained: its behavior depends on state it does not control. This makes testing, debugging, and parallelization difficult. Pass data via parameters and return values instead.
Wrapping Up
Functions are the building blocks of clean, maintainable code. They isolate logic, enable reuse, and make testing possible. Master the parameter types (positional, keyword, default, *args, **kwargs), understand scope (LEGB), avoid the mutable default trap, and embrace first-class functions for flexible designs. In data engineering, every pipeline step should be a function — readable, testable, and composable.
Previous in this series: Loops — for, while, enumerate, zip, break/continue
Next in this series: List Comprehensions, Dict Comprehensions, and Generator Expressions
Naveen Vuppula is a Senior Data Engineering Consultant and app developer based in Ontario, Canada. He writes about Python, SQL, AWS, Azure, and everything data engineering at DriveDataScience.com.