Python Comprehensions and Generator Expressions: List, Dict, Set Comprehensions, Generator Expressions, Nested Comprehensions, Walrus Operator, and Performance Patterns

In the Loops post, we saw append inside a for loop to build a list. In the Lists post, we introduced list comprehensions as a faster, cleaner alternative. This post goes all the way — every type of comprehension, when to use each, when NOT to use them, and the powerful generator expressions that process data without loading everything into memory.

Comprehensions are one of Python’s most loved features. They let you express an entire loop + condition + transformation in a single, readable line. Once you master them, you will never go back to writing three-line append loops for simple transformations.

Think of a comprehension like a conveyor belt in a factory. Raw materials (input iterable) go in one end. Each item passes through a quality check (filter condition) and a processing station (transformation expression). Finished products come out the other end as a new collection. The entire factory is described in one line instead of a multi-page manual (loop with append).

Table of Contents

• Why Comprehensions Exist
• The Loop-and-Append Pattern They Replace
• List Comprehensions
• Basic Syntax
• With Condition (Filtering)
• With if-else (Transformation)
• The Position Rule: if After vs if-else Before
• Multiple Conditions
• Calling Functions in Comprehensions
• Dict Comprehensions
• Basic Dict Comprehension
• Filtering a Dict
• Transforming Keys or Values
• Inverting a Dict
• Building Lookup Tables
• Set Comprehensions
• Basic Set Comprehension
• Deduplication with Transformation
• Nested Comprehensions
• Flattening a 2D List
• Creating a 2D List
• Cartesian Product
• Reading Order (Left to Right)
• When Nested Becomes Unreadable
• Generator Expressions
• What Is a Generator Expression
• Why Generators Use Almost No Memory
• Generator vs List Comprehension
• Generators Are Single-Use
• Using Generators with Built-in Functions
• Generator Expressions in Function Calls
• Walrus Operator in Comprehensions (Python 3.8+)
• Avoiding Repeated Computation
• Comprehension Anti-Patterns
• When NOT to Use Comprehensions
• Side Effects in Comprehensions
• Too-Long Comprehensions
• Performance Comparison
• List Comprehension vs Loop vs map()
• Generator vs List for Large Data
• Real-World Data Engineering Patterns
• Cleaning Column Names
• Building SQL Queries Dynamically
• Processing Config Files
• Filtering and Transforming Records
• Common Mistakes
• Interview Questions
• Wrapping Up

Why Comprehensions Exist

The Loop-and-Append Pattern They Replace

# The old way: 4 lines to build a filtered, transformed list
results = []
for x in range(10):
    if x % 2 == 0:
        results.append(x ** 2)
print(results)  # [0, 4, 16, 36, 64]

# The comprehension way: 1 line, same result
results = [x ** 2 for x in range(10) if x % 2 == 0]
print(results)  # [0, 4, 16, 36, 64]

# Both produce EXACTLY the same output.
# The comprehension is:
#   - Shorter (1 line vs 4)
#   - Faster (20-50% in CPython)
#   - More Pythonic (the "expected" way in Python culture)
#   - Easier to read (once you learn the syntax)

Real-life analogy: The loop-and-append pattern is like manually writing each item on a shopping list one by one — “check if I need milk, write it down, check if I need eggs, write it down.” A comprehension is like saying “write down everything from the grocery aisle that I need” — one instruction that covers the whole process.

List Comprehensions

Basic Syntax

# Syntax: [expression FOR variable IN iterable]

# Square every number
squares = [x ** 2 for x in range(6)]
print(squares)  # [0, 1, 4, 9, 16, 25]

# Uppercase every name
names = ["alice", "bob", "charlie"]
upper = [name.upper() for name in names]
print(upper)  # ['ALICE', 'BOB', 'CHARLIE']

# Get lengths
lengths = [len(name) for name in names]
print(lengths)  # [5, 3, 7]

# Convert types
str_nums = ["10", "20", "30", "40"]
int_nums = [int(s) for s in str_nums]
print(int_nums)  # [10, 20, 30, 40]

# Extract first character
initials = [name[0] for name in names]
print(initials)  # ['a', 'b', 'c']

With Condition (Filtering)

# Syntax: [expression FOR variable IN iterable IF condition]

# Only even numbers
evens = [x for x in range(20) if x % 2 == 0]
print(evens)  # [0, 2, 4, 6, 8, 10, 12, 14, 16, 18]

# Only long names
long_names = [name for name in names if len(name) > 4]
print(long_names)  # ['alice', 'charlie']

# Only positive numbers
nums = [5, -3, 8, -1, 0, 12, -7]
positives = [n for n in nums if n > 0]
print(positives)  # [5, 8, 12]

# Only non-null values
data = ["hello", None, "world", None, "python"]
clean = [val for val in data if val is not None]
print(clean)  # ['hello', 'world', 'python']

# Filter AND transform in one step
upper_long = [name.upper() for name in names if len(name) > 4]
print(upper_long)  # ['ALICE', 'CHARLIE']

With if-else (Transformation)

# Syntax: [true_expr IF condition ELSE false_expr FOR variable IN iterable]

# Label each number
labels = ["even" if x % 2 == 0 else "odd" for x in range(6)]
print(labels)  # ['even', 'odd', 'even', 'odd', 'even', 'odd']

# Replace negatives with zero
nums = [5, -3, 8, -1, 0, 12, -7]
clamped = [n if n >= 0 else 0 for n in nums]
print(clamped)  # [5, 0, 8, 0, 0, 12, 0]

# Categorize ages
ages = [5, 15, 25, 65, 80]
categories = ["minor" if a < 18 else "senior" if a >= 65 else "adult" for a in ages]
print(categories)  # ['minor', 'minor', 'adult', 'senior', 'senior']

# Pass/fail
scores = [88, 42, 95, 67, 55]
results = ["pass" if s >= 60 else "fail" for s in scores]
print(results)  # ['pass', 'fail', 'pass', 'pass', 'fail']

The Position Rule: if After vs if-else Before

This is the most confusing part of comprehensions. Here is the definitive rule:

  FILTERING (keep or discard items):
    [expr for x in items IF condition]
    The IF goes AFTER the for — it decides which items to INCLUDE
    Example: [x for x in nums if x > 0]  → keeps only positives

  TRANSFORMING (change every item):
    [true_val IF condition ELSE false_val for x in items]
    The IF-ELSE goes BEFORE the for — it decides which VALUE to produce
    Example: [x if x > 0 else 0 for x in nums]  → replaces negatives with 0

  Memory trick:
    IF alone (filter) → goes at the END: [...for x in items IF...]
    IF-ELSE (transform) → goes at the START: [IF...ELSE... for x in items]

  You CANNOT have if-else at the END — that is a syntax error:
    [x for x in items if x > 0 else 0]  → SyntaxError!

Multiple Conditions

# Multiple conditions with and/or
nums = range(100)

# Divisible by both 3 AND 5
fizzbuzz = [n for n in nums if n % 3 == 0 and n % 5 == 0]
print(fizzbuzz)  # [0, 15, 30, 45, 60, 75, 90]

# Between 10 and 50 OR greater than 90
selected = [n for n in nums if (10 <= n <= 50) or n > 90]

# Multiple filters (same as AND — reads like stacked conditions)
result = [x for x in range(100) if x % 2 == 0 if x % 3 == 0]
# Same as: [x for x in range(100) if x % 2 == 0 and x % 3 == 0]
print(result)  # [0, 6, 12, 18, 24, 30, ...]

Calling Functions in Comprehensions

# Any function call works in the expression or condition
def is_valid_email(email):
    return email and "@" in email and "." in email

emails = ["alice@email.com", "bad-email", None, "bob@corp.org", ""]
valid_emails = [e for e in emails if is_valid_email(e)]
print(valid_emails)  # ['alice@email.com', 'bob@corp.org']

# Clean and filter in one pass
def clean_name(name):
    return name.strip().title()

raw_names = ["  alice ", "BOB", "  charlie  ", "", "  diana"]
clean_names = [clean_name(n) for n in raw_names if n.strip()]
print(clean_names)  # ['Alice', 'Bob', 'Charlie', 'Diana']

Dict Comprehensions

Basic Dict Comprehension

# Syntax: {key_expr: value_expr FOR variable IN iterable}

# Number → square mapping
squares = {n: n ** 2 for n in range(6)}
print(squares)  # {0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25}

# Name → length mapping
names = ["alice", "bob", "charlie"]
name_lengths = {name: len(name) for name in names}
print(name_lengths)  # {'alice': 5, 'bob': 3, 'charlie': 7}

# From two lists using zip
keys = ["name", "age", "city"]
values = ["Naveen", 30, "Toronto"]
person = {k: v for k, v in zip(keys, values)}
print(person)  # {'name': 'Naveen', 'age': 30, 'city': 'Toronto'}

Filtering a Dict

# Keep only certain key-value pairs
config = {
    "db_host": "server.com",
    "db_port": 1433,
    "api_key": "secret123",
    "api_url": "https://api.example.com",
    "log_level": "INFO"
}

# Only database settings
db_config = {k: v for k, v in config.items() if k.startswith("db_")}
print(db_config)  # {'db_host': 'server.com', 'db_port': 1433}

# Only string values
string_vals = {k: v for k, v in config.items() if isinstance(v, str)}
print(string_vals)  # {'db_host': 'server.com', 'api_key': '...', ...}

# Remove null values
data = {"name": "Naveen", "email": None, "city": "Toronto", "phone": None}
clean = {k: v for k, v in data.items() if v is not None}
print(clean)  # {'name': 'Naveen', 'city': 'Toronto'}

Transforming Keys or Values

# Lowercase all keys
messy = {"Name": "Naveen", "AGE": 30, "City": "Toronto"}
clean = {k.lower(): v for k, v in messy.items()}
print(clean)  # {'name': 'Naveen', 'age': 30, 'city': 'Toronto'}

# Add prefix to all keys
prefixed = {f"user_{k}": v for k, v in clean.items()}
print(prefixed)  # {'user_name': 'Naveen', 'user_age': 30, 'user_city': 'Toronto'}

# Rename keys using a mapping
old_data = {"firstName": "Naveen", "lastName": "Vuppula"}
rename_map = {"firstName": "first_name", "lastName": "last_name"}
new_data = {rename_map.get(k, k): v for k, v in old_data.items()}
print(new_data)  # {'first_name': 'Naveen', 'last_name': 'Vuppula'}

# Transform values (uppercase strings, leave others unchanged)
record = {"name": "naveen", "age": 30, "city": "toronto"}
upper_strs = {k: v.upper() if isinstance(v, str) else v for k, v in record.items()}
print(upper_strs)  # {'name': 'NAVEEN', 'age': 30, 'city': 'TORONTO'}

Inverting a Dict

# Swap keys and values
codes = {"US": "United States", "CA": "Canada", "IN": "India"}
inverted = {v: k for k, v in codes.items()}
print(inverted)  # {'United States': 'US', 'Canada': 'CA', 'India': 'IN'}

# WARNING: if values are not unique, later entries OVERWRITE earlier ones
# grades = {"Alice": "A", "Bob": "B", "Charlie": "A"}
# inverted = {v: k for k, v in grades.items()}
# {'A': 'Charlie', 'B': 'Bob'} — Alice is LOST because Charlie also has 'A'

# Safe inversion for non-unique values: group into lists
from collections import defaultdict
grades = {"Alice": "A", "Bob": "B", "Charlie": "A", "Diana": "B"}
inverted = defaultdict(list)
for name, grade in grades.items():
    inverted[grade].append(name)
print(dict(inverted))  # {'A': ['Alice', 'Charlie'], 'B': ['Bob', 'Diana']}

Building Lookup Tables

# Convert a list of records to an O(1) lookup by a key field
employees = [
    {"id": 101, "name": "Alice", "dept": "Engineering"},
    {"id": 102, "name": "Bob", "dept": "Marketing"},
    {"id": 103, "name": "Charlie", "dept": "Engineering"},
]

# Lookup by ID
by_id = {emp["id"]: emp for emp in employees}
print(by_id[102]["name"])  # "Bob" — O(1) access

# Group by department
from collections import defaultdict
by_dept = defaultdict(list)
for emp in employees:
    by_dept[emp["dept"]].append(emp["name"])
print(dict(by_dept))  # {'Engineering': ['Alice', 'Charlie'], 'Marketing': ['Bob']}

Set Comprehensions

Basic Set Comprehension

# Syntax: {expression FOR variable IN iterable}
# Same as list comprehension but with {} — produces a SET (unique values, unordered)

# Unique squares
squares = {x ** 2 for x in range(-3, 4)}
print(squares)  # {0, 1, 4, 9} — duplicates removed ((-3)^2 = 3^2 = 9)

# Unique file extensions
files = ["data.csv", "report.pdf", "sales.csv", "image.png", "log.csv"]
extensions = {f.rsplit(".", 1)[-1] for f in files}
print(extensions)  # {'csv', 'pdf', 'png'} — no duplicate 'csv'

# Unique words in a text
text = "the cat sat on the mat and the cat slept"
unique_words = {word for word in text.split()}
print(unique_words)  # {'the', 'cat', 'sat', 'on', 'mat', 'and', 'slept'}

Deduplication with Transformation

# Clean AND deduplicate in one pass
raw_emails = [" Alice@Email.COM ", "bob@email.com", "ALICE@email.com", "  bob@email.com  "]
clean_unique = {email.strip().lower() for email in raw_emails}
print(clean_unique)  # {'alice@email.com', 'bob@email.com'} — cleaned + deduplicated

# Unique first characters
names = ["Alice", "Anna", "Bob", "Charlie", "Catherine"]
first_chars = {name[0] for name in names}
print(first_chars)  # {'A', 'B', 'C'} — only 3 unique initials from 5 names

Nested Comprehensions

Flattening a 2D List

# Convert a list of lists into a single flat list
matrix = [[1, 2, 3], [4, 5, 6], [7, 8, 9]]

# Loop version:
flat = []
for row in matrix:
    for num in row:
        flat.append(num)

# Comprehension version (same result):
flat = [num for row in matrix for num in row]
print(flat)  # [1, 2, 3, 4, 5, 6, 7, 8, 9]

# With filter: only even numbers from the matrix
even_flat = [num for row in matrix for num in row if num % 2 == 0]
print(even_flat)  # [2, 4, 6, 8]

Creating a 2D List

# Create a 3x4 matrix of zeros
zeros = [[0 for col in range(4)] for row in range(3)]
print(zeros)  # [[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]

# REMEMBER: [[0]*4]*3 is WRONG — all rows share the same list object!
# Always use the comprehension form for 2D lists

# Multiplication table
table = [[i * j for j in range(1, 6)] for i in range(1, 6)]
for row in table:
    print(row)
# [1, 2, 3, 4, 5]
# [2, 4, 6, 8, 10]
# [3, 6, 9, 12, 15]
# [4, 8, 12, 16, 20]
# [5, 10, 15, 20, 25]

# Identity matrix
n = 4
identity = [[1 if i == j else 0 for j in range(n)] for i in range(n)]
# [[1,0,0,0], [0,1,0,0], [0,0,1,0], [0,0,0,1]]

Cartesian Product

# All combinations of two sequences
colors = ["red", "blue"]
sizes = ["S", "M", "L"]
combos = [(color, size) for color in colors for size in sizes]
print(combos)
# [('red', 'S'), ('red', 'M'), ('red', 'L'),
#  ('blue', 'S'), ('blue', 'M'), ('blue', 'L')]

# Practical: generate all test case combinations
envs = ["dev", "staging", "prod"]
formats = ["csv", "parquet", "delta"]
test_cases = [(env, fmt) for env in envs for fmt in formats]
print(f"{len(test_cases)} test cases")  # 9 test cases

Reading Order (Left to Right)

Nested comprehension reading order = same as nested loop order:

  COMPREHENSION:  [num for row in matrix for num in row if num > 0]

  EQUIVALENT LOOP:
    for row in matrix:          ← first "for" in comprehension
        for num in row:         ← second "for" in comprehension
            if num > 0:         ← "if" in comprehension
                result.append(num)  ← expression at the START

  Read left to right: for row → for num → if condition → expression

When Nested Becomes Unreadable

# ❌ TOO COMPLEX — three levels of nesting in a comprehension
result = [cell for table in database for row in table for cell in row if cell is not None]

# ✅ BETTER — use a regular loop when it is more than two levels deep
result = []
for table in database:
    for row in table:
        for cell in row:
            if cell is not None:
                result.append(cell)

# RULE OF THUMB:
# 1 for loop → always use a comprehension
# 2 for loops → use a comprehension if it fits on one readable line
# 3+ for loops → use a regular loop (readability wins)

Generator Expressions

What Is a Generator Expression

# A generator expression looks like a list comprehension but with () instead of []
# It produces values ONE AT A TIME instead of creating the entire list in memory

# List comprehension: creates ALL values immediately, stores in memory
squares_list = [x ** 2 for x in range(1000000)]   # 1 million items in memory NOW

# Generator expression: creates values ON DEMAND, stores almost nothing
squares_gen = (x ** 2 for x in range(1000000))     # Nothing computed yet!

print(type(squares_list))  # <class 'list'>
print(type(squares_gen))   # <class 'generator'>

# Access values one at a time
print(next(squares_gen))   # 0 — first value computed now
print(next(squares_gen))   # 1 — second value computed now
print(next(squares_gen))   # 4 — third value computed now
# The remaining 999,997 values have NOT been computed yet!

Real-life analogy: A list comprehension is like printing an entire phone book — every number is printed upfront, and the book takes up shelf space. A generator expression is like a directory assistance operator — you call, ask for ONE number, get it, and the operator waits for your next call. No phone book needed. If you only need 10 numbers out of a million, you save enormous resources.

Why Generators Use Almost No Memory

import sys

# List: stores ALL 1 million values
list_comp = [x ** 2 for x in range(1_000_000)]
print(f"List: {sys.getsizeof(list_comp):,} bytes")   # ~8,448,728 bytes (8 MB)

# Generator: stores only the FORMULA, not the values
gen_expr = (x ** 2 for x in range(1_000_000))
print(f"Generator: {sys.getsizeof(gen_expr):,} bytes")  # ~200 bytes!

# 8 MB vs 200 bytes — the generator is ~42,000x smaller
# Because it computes each value only when requested (lazy evaluation)

Generator vs List Comprehension

Feature	List Comprehension [...]	Generator Expression (...)
Memory	Stores ALL values	Stores almost nothing (lazy)
Speed to first value	Slow (computes all first)	Instant (computes one)
Reusable?	Yes (iterate multiple times)	No (single-use — exhausted after one pass)
Indexing?	Yes (list[3])	No (only next() or for)
len()?	Yes	No
Use when	You need the full list, or will iterate multiple times	You process items one at a time, or data is huge

Generators Are Single-Use

# WARNING: generators are EXHAUSTED after one pass
gen = (x for x in range(5))

print(list(gen))  # [0, 1, 2, 3, 4] — consumed all values
print(list(gen))  # [] — EMPTY! The generator is exhausted

# This catches people off guard:
gen = (x ** 2 for x in range(5))
print(sum(gen))    # 30 — consumes the generator
print(sum(gen))    # 0 — generator is empty, sum of nothing = 0

# If you need to iterate multiple times, use a list comprehension instead
# Or re-create the generator expression each time

Using Generators with Built-in Functions

# Built-in functions like sum(), min(), max(), any(), all() consume generators efficiently
# They process one value at a time — no need to build the full list

# Sum of squares (no intermediate list created)
total = sum(x ** 2 for x in range(1_000_000))
print(total)  # 333332833333500000

# Find maximum length (no list of lengths created)
names = ["alice", "bob", "charlie", "diana"]
max_len = max(len(name) for name in names)
print(max_len)  # 7

# Check if any name is too long
has_long = any(len(name) > 10 for name in names)
print(has_long)  # False — stops at the first True (short-circuit!)

# Check if all names are non-empty
all_valid = all(len(name) > 0 for name in names)
print(all_valid)  # True

# Count items matching a condition (sum of booleans)
error_count = sum(1 for line in log_lines if "ERROR" in line)
# OR: sum("ERROR" in line for line in log_lines) — True = 1

Generator Expressions in Function Calls

# When a generator is the ONLY argument, you can drop the extra parentheses
# sum([x**2 for x in range(10)])  — list comprehension (creates list, then sums)
# sum((x**2 for x in range(10)))  — generator (no list, sums one by one)
# sum(x**2 for x in range(10))    — same as above (parentheses from sum() suffice)

# All three produce the same result, but the generator versions use less memory

# More examples:
",".join(str(x) for x in range(5))          # "0,1,2,3,4"
sorted(name.lower() for name in names)       # sorted generator
min(len(word) for word in words)             # min of lengths
max(order["amount"] for order in orders)     # max of amounts

Walrus Operator in Comprehensions (Python 3.8+)

Avoiding Repeated Computation

# The walrus operator := assigns and returns a value in one expression
# Useful in comprehensions when you need to compute a value for BOTH filtering and output

# ❌ WITHOUT walrus — function called TWICE per item
results = [process(x) for x in data if process(x) is not None]
# process(x) runs once for the if check, then AGAIN for the value — wasteful!

# ✅ WITH walrus — function called ONCE per item
results = [y for x in data if (y := process(x)) is not None]
# y := process(x) computes the result, assigns it to y, AND returns it for the if check
# The same y is then used as the value — no double computation

# Practical: filter and keep cleaned values
raw_names = ["  alice  ", "", "  bob  ", "  ", "  charlie  "]
clean = [cleaned for name in raw_names if (cleaned := name.strip())]
print(clean)  # ['alice', 'bob', 'charlie'] — empty strings filtered out

# Without walrus, you would call strip() twice:
# clean = [name.strip() for name in raw_names if name.strip()]

# Practical: compute expensive value once
import re
pattern = re.compile(r'\d{3}-\d{3}-\d{4}')
texts = ["Call 416-555-1234", "No phone here", "Reach us at 905-123-4567"]
phones = [m.group() for text in texts if (m := pattern.search(text))]
print(phones)  # ['416-555-1234', '905-123-4567']

Comprehension Anti-Patterns

When NOT to Use Comprehensions

# ❌ When you do not need the result (side effects only)
# BAD: [print(x) for x in range(5)]  — creates a list of None values!
# GOOD: for x in range(5): print(x)

# ❌ When the logic is complex (multiple statements per item)
# BAD:
# result = [
#     (x, y, x*y, "big" if x*y > 100 else "small")
#     for x in range(20)
#     for y in range(20)
#     if x > 0 and y > 0 and x != y and (x*y) % 7 == 0
# ]
# GOOD: use a regular loop for anything this complex

# ❌ When you are already using PySpark
# Comprehensions are Python-side operations — they do NOT parallelize
# Use PySpark DataFrame operations (filter, withColumn) for distributed data

Side Effects in Comprehensions

# Comprehensions should PRODUCE values, not perform actions

# ❌ Using comprehension for side effects (print, write, API calls)
[print(x) for x in range(5)]           # Works but produces a list of None — waste
[send_email(user) for user in users]    # Works but the list is thrown away

# ✅ Use a regular loop for side effects
for x in range(5):
    print(x)

for user in users:
    send_email(user)

# RULE: if you are not keeping the result, do not use a comprehension

Too-Long Comprehensions

# If a comprehension does not fit on one line comfortably, break it up

# ✅ Readable multi-line comprehension (when necessary)
results = [
    transform(item)
    for item in source_data
    if item.is_valid()
    if item.status == "active"
]

# ✅ Or use a named function for complex logic
def process_item(item):
    cleaned = clean(item)
    validated = validate(cleaned)
    return transform(validated) if validated else None

results = [r for item in data if (r := process_item(item)) is not None]

Performance Comparison

List Comprehension vs Loop vs map()

import timeit

data = list(range(1_000_000))

# Method 1: Loop + append
def loop_method():
    result = []
    for x in data:
        result.append(x ** 2)
    return result

# Method 2: List comprehension
def comp_method():
    return [x ** 2 for x in data]

# Method 3: map()
def map_method():
    return list(map(lambda x: x ** 2, data))

# Benchmark
print(f"Loop:          {timeit.timeit(loop_method, number=10):.3f}s")
print(f"Comprehension: {timeit.timeit(comp_method, number=10):.3f}s")
print(f"map():         {timeit.timeit(map_method, number=10):.3f}s")

# Typical results:
# Loop:          3.200s
# Comprehension: 2.100s  ← ~35% faster than loop
# map():         2.500s  ← between loop and comprehension

# Comprehension wins because:
# 1. No .append() method lookup on each iteration
# 2. Optimized C-level bytecode in CPython
# 3. Pre-allocates memory for the result list

Generator vs List for Large Data

# When you only need to iterate ONCE, generators save massive memory

# ❌ Creates 10 million items in memory, then sums
total = sum([x ** 2 for x in range(10_000_000)])  # ~80 MB of memory

# ✅ Creates values one at a time, sums on the fly
total = sum(x ** 2 for x in range(10_000_000))    # ~200 bytes of memory

# Same result. Same speed. But 400,000x less memory.

Real-World Data Engineering Patterns

Cleaning Column Names

# Standardize column names from a messy source
raw_columns = ["Customer ID", "First Name", "Last Name", "E-Mail Address"]

# snake_case conversion
clean_columns = [col.lower().replace(" ", "_").replace("-", "_") for col in raw_columns]
print(clean_columns)  # ['customer_id', 'first_name', 'last_name', 'e_mail_address']

# In PySpark:
# for old, new in zip(raw_columns, clean_columns):
#     df = df.withColumnRenamed(old, new)

Building SQL Queries Dynamically

# Build SELECT with only non-null columns
columns = ["customer_id", "name", "email", "phone", "city"]
available = [col for col in columns if col in source_schema]

select_clause = ", ".join(available)
sql = f"SELECT {select_clause} FROM customers WHERE active = 1"
print(sql)

# Build CASE WHEN for categorization
categories = {"small": 100, "medium": 1000, "large": 10000}
case_parts = [f"WHEN amount <= {threshold} THEN '{label}'" 
              for label, threshold in categories.items()]
case_sql = f"CASE {' '.join(case_parts)} ELSE 'xlarge' END AS size_category"

Processing Config Files

# Parse a simple key=value config file
config_lines = [
    "# Database settings",
    "db_host=server.database.windows.net",
    "db_port=1433",
    "",
    "# API settings",
    "api_key=secret123",
    "api_timeout=30",
]

# Filter comments and empty lines, then split into key-value pairs
config = dict(
    line.split("=", 1)
    for line in config_lines
    if line.strip() and not line.startswith("#")
)
print(config)
# {'db_host': 'server.database.windows.net', 'db_port': '1433',
#  'api_key': 'secret123', 'api_timeout': '30'}

Filtering and Transforming Records

# Process a batch of records
records = [
    {"id": 1, "name": "Alice", "status": "active", "score": 88},
    {"id": 2, "name": "Bob", "status": "inactive", "score": 42},
    {"id": 3, "name": "Charlie", "status": "active", "score": 95},
    {"id": 4, "name": "Diana", "status": "active", "score": 67},
]

# Active users with passing scores
passing_active = [
    {"id": r["id"], "name": r["name"], "grade": "A" if r["score"] >= 90 else "B" if r["score"] >= 80 else "C"}
    for r in records
    if r["status"] == "active" and r["score"] >= 60
]
print(passing_active)
# [{'id': 1, 'name': 'Alice', 'grade': 'B'},
#  {'id': 3, 'name': 'Charlie', 'grade': 'A'},
#  {'id': 4, 'name': 'Diana', 'grade': 'C'}]

Common Mistakes

1. Confusing the if position (filter vs transform) — [x for x in items if cond] FILTERS (keeps matching items). [a if cond else b for x in items] TRANSFORMS (changes every item). Mixing them up causes SyntaxError or wrong results.
2. Using comprehensions for side effects — [print(x) for x in items] creates a useless list of None values. Use a regular for loop for actions like printing, writing, or sending requests.
3. Iterating a generator twice — generators are exhausted after one pass. gen = (x for x in data); list(gen); list(gen) → second call returns []. Use a list comprehension if you need multiple passes.
4. Using list comprehension when a generator suffices — sum([x**2 for x in range(10_000_000)]) creates a 80 MB list just to sum it. sum(x**2 for x in range(10_000_000)) uses 200 bytes. Drop the brackets when passing to functions like sum, min, max, any, all, join.
5. Writing too-complex comprehensions — if it has 3+ nested loops, multiple conditions, and a complex expression, use a regular loop. Comprehensions should be READABLE, not clever.
6. Calling expensive functions twice (filter + expression) — [f(x) for x in data if f(x) > 0] calls f(x) twice per item. Use the walrus operator: [y for x in data if (y := f(x)) > 0].

Interview Questions

Q: What is the difference between a list comprehension and a generator expression? A: A list comprehension [...] creates the entire list in memory immediately. A generator expression (...) produces values lazily, one at a time, using almost no memory. Use list comprehensions when you need the full list or will iterate multiple times. Use generators for large datasets or when passing to functions like sum() that consume values one by one.

Q: Where does the if go in a list comprehension — before or after the for? A: if alone (filtering) goes AFTER the for: [x for x in items if condition]. if-else (transforming) goes BEFORE the for: [a if condition else b for x in items]. The position determines whether you are filtering items out or transforming every item.

Q: Why are list comprehensions faster than loop-and-append? A: Comprehensions are optimized at the C level in CPython. They avoid the overhead of looking up and calling the .append() method on every iteration, pre-allocate memory for the result, and use a specialized bytecode instruction. Typically 20-50% faster for simple transformations.

Q: What is the walrus operator and how is it used in comprehensions? A: The walrus operator := (Python 3.8+) assigns a value to a variable as part of an expression. In comprehensions, it avoids calling a function twice: [y for x in data if (y := func(x)) is not None]. The function runs once, the result is assigned to y for the filter check AND used as the output value.

Q: When should you NOT use a comprehension? A: When performing side effects (printing, writing files, API calls) — use a loop. When the logic is complex (3+ nested loops, multiple conditions, multi-step processing) — use a loop for readability. When working with PySpark — use DataFrame operations for distributed processing, not Python-side comprehensions.

Q: What is a dict comprehension and give a practical use case? A: {key: value for item in iterable} creates a dictionary. Practical use: building a lookup table from a list of records: lookup = {emp["id"]: emp for emp in employees}. This converts O(n) list searches to O(1) dict lookups — critical for performance in data pipelines processing millions of records.

Wrapping Up

Comprehensions are Python at its most expressive — an entire loop, filter, and transformation in one readable line. List comprehensions for building lists, dict comprehensions for building lookups, set comprehensions for unique values, and generator expressions for memory-efficient streaming. Master the position rule (if-after for filtering, if-else-before for transforming), know when to use generators over lists (large data, single-pass), and resist the urge to make comprehensions too complex.

In data engineering, comprehensions shine for cleaning column names, building SQL dynamically, parsing configs, and transforming records. But for distributed processing at scale, let PySpark handle it — comprehensions are Python-side, single-machine operations.

Previous in this series: Functions — Parameters, Return Values, Scope, *args, **kwargs

Next in this series: Lambda Functions, map(), filter(), reduce()

---

Naveen Vuppula is a Senior Data Engineering Consultant and app developer based in Ontario, Canada. He writes about Python, SQL, AWS, Azure, and everything data engineering at DriveDataScience.com.