Python Dictionaries: Nesting, Comprehensions, DefaultDict, Counter, Merging, Iteration Patterns, and Why Dicts Power Everything in Python

If lists are Python’s workhorse, dictionaries are its brain. Every namespace is a dictionary. Every object’s attributes live in a dictionary (__dict__). Every JSON response from an API is parsed into a dictionary. Every keyword argument in a function call passes through a dictionary. Dictionaries are so fundamental to Python that the entire language is built on them.

Yet most tutorials teach dictionaries in 5 minutes: “key-value pairs, use square brackets to access.” That misses the power. Nested dicts for hierarchical data. DefaultDict for counting and grouping without KeyError. Counter for frequency analysis. Dict comprehensions for transformations. Merging strategies for combining configs. This post covers all of it.

Think of a dictionary like a real-world filing cabinet. Each drawer has a label (key) and contains a folder (value). You do not search drawer by drawer — you go directly to the label you need. That is why dict lookup is O(1) — the label tells you exactly where to look. Lists are like a stack of unmarked folders — you must flip through every one to find what you need.

Table of Contents

• Creating Dictionaries
• Multiple Ways to Create a Dict
• The Empty Dict
• Dictionaries in Memory
• Hash Tables Under the Hood
• Why Keys Must Be Immutable
• Accessing Values
• Square Bracket Access
• get() — Safe Access with Default
• Why get() Should Be Your Default
• Adding, Updating, and Removing
• Adding and Updating Key-Value Pairs
• update() — Merge Another Dict
• Removing Keys
• setdefault() — Get or Set
• Iterating Over Dictionaries
• keys(), values(), items()
• Iteration Patterns for Data Engineering
• Unpacking Dicts in Loops
• Dictionary Comprehensions
• Basic Comprehension
• Comprehension with Condition
• Inverting a Dictionary
• Transforming Keys or Values
• Nested Dictionaries
• Creating and Accessing Nested Dicts
• Safe Nested Access (Avoiding KeyError Chains)
• Modifying Nested Dicts
• Flattening Nested Dicts
• DefaultDict — Never Get a KeyError Again
• defaultdict(int) — Counting
• defaultdict(list) — Grouping
• defaultdict(set) — Unique Grouping
• defaultdict vs setdefault
• Counter — Frequency Analysis
• Creating a Counter
• most_common()
• Counter Arithmetic
• OrderedDict and ChainMap
• Merging Dictionaries
• update() Method
• Unpacking with **
• Merge Operator | (Python 3.9+)
• Which Values Win in a Merge
• Dictionaries as Function Arguments
• **kwargs
• Unpacking Dicts into Function Calls
• Real-World Data Engineering Patterns
• Config Files and Settings
• JSON to Dict and Back
• Building Lookup Tables
• Grouping and Aggregating Data
• Common Mistakes
• Interview Questions
• Wrapping Up

Creating Dictionaries

Multiple Ways to Create a Dict

# 1. Curly braces with key: value pairs
student = {"name": "Naveen", "age": 30, "city": "Toronto"}

# 2. dict() constructor with keyword arguments
student = dict(name="Naveen", age=30, city="Toronto")

# 3. dict() from a list of tuples
student = dict([("name", "Naveen"), ("age", 30), ("city", "Toronto")])

# 4. dict.fromkeys() — same value for multiple keys
statuses = dict.fromkeys(["pending", "active", "closed"], 0)
print(statuses)  # {'pending': 0, 'active': 0, 'closed': 0}

# 5. Dict comprehension
squares = {x: x**2 for x in range(6)}
print(squares)   # {0: 0, 1: 1, 2: 4, 3: 9, 4: 16, 5: 25}

# 6. zip() two lists into a dict
keys = ["name", "age", "city"]
values = ["Naveen", 30, "Toronto"]
student = dict(zip(keys, values))
print(student)   # {'name': 'Naveen', 'age': 30, 'city': 'Toronto'}

The Empty Dict

# Two ways to create an empty dict
empty1 = {}
empty2 = dict()
print(type(empty1))  # <class 'dict'>

# Remember from the Sets post: {} is a dict, NOT a set!
# Use set() for an empty set

Dictionaries in Memory

Hash Tables Under the Hood

How Python dicts work internally:

  student = {"name": "Naveen", "age": 30, "city": "Toronto"}

  Step 1: hash("name") → 12345
  Step 2: 12345 % table_size → slot 5
  Step 3: Store ("name", "Naveen") in slot 5

  Step 1: hash("age") → 67890
  Step 2: 67890 % table_size → slot 2
  Step 3: Store ("age", 30) in slot 2

  Looking up student["city"]:
    hash("city") → 11111
    11111 % table_size → slot 3
    Check slot 3 → ("city", "Toronto") → return "Toronto"

  ONE step to find any key, regardless of dict size = O(1)

  Hash table (simplified):
  ┌────────┬─────────────────────────┐
  │ Slot 0 │ (empty)                 │
  │ Slot 1 │ (empty)                 │
  │ Slot 2 │ ("age", 30)             │
  │ Slot 3 │ ("city", "Toronto")     │
  │ Slot 4 │ (empty)                 │
  │ Slot 5 │ ("name", "Naveen")      │
  │ Slot 6 │ (empty)                 │
  │ Slot 7 │ (empty)                 │
  └────────┴─────────────────────────┘

Why Keys Must Be Immutable

Dict keys must be hashable (immutable). If a key could change after insertion, its hash would change, and Python would look in the wrong slot — the key would be lost forever. This is why strings, numbers, tuples, and frozensets can be keys, but lists, sets, and dicts cannot.

# Valid keys (immutable / hashable)
d = {
    "name": "Naveen",           # str ✓
    42: "answer",               # int ✓
    3.14: "pi",                 # float ✓
    (1, 2): "coordinates",      # tuple ✓
    True: "boolean",            # bool ✓
    frozenset({1, 2}): "fset",  # frozenset ✓
}

# Invalid keys (mutable / unhashable)
# d[[1, 2]] = "list key"       # TypeError: unhashable type: 'list'
# d[{1, 2}] = "set key"        # TypeError: unhashable type: 'set'
# d[{"a": 1}] = "dict key"     # TypeError: unhashable type: 'dict'

Accessing Values

Square Bracket Access

student = {"name": "Naveen", "age": 30, "city": "Toronto"}

print(student["name"])    # "Naveen"
print(student["age"])     # 30

# KeyError if key does not exist
# print(student["email"])  # KeyError: 'email'

get() — Safe Access with Default

# get() returns None instead of raising KeyError
print(student.get("email"))           # None (no error!)
print(student.get("email", "N/A"))    # "N/A" (custom default)
print(student.get("name", "Unknown")) # "Naveen" (key exists, returns actual value)

# Practical: safely access nested API response fields
response = {"data": {"user": {"name": "Naveen"}}}
name = response.get("data", {}).get("user", {}).get("name", "Unknown")
print(name)  # "Naveen"

# If any level is missing, returns "Unknown" instead of crashing

Why get() Should Be Your Default

Use [] when the key MUST exist (absence is a bug). Use get() when the key MIGHT not exist (absence is normal). In data engineering, API responses and config files often have missing keys — get() prevents crashes.

Real-life analogy: student["email"] is like demanding “give me the file from drawer E!” — if there is no drawer E, the assistant panics (KeyError). student.get("email", "N/A") is like asking “do we have a file in drawer E? If not, just say N/A.” Much more graceful.

Adding, Updating, and Removing

Adding and Updating Key-Value Pairs

student = {"name": "Naveen", "age": 30}

# Add a new key
student["email"] = "naveen@email.com"
print(student)  # {'name': 'Naveen', 'age': 30, 'email': 'naveen@email.com'}

# Update an existing key (same syntax)
student["age"] = 31
print(student)  # {'name': 'Naveen', 'age': 31, 'email': 'naveen@email.com'}

# Python does not distinguish between "add" and "update"
# If the key exists → update. If it does not → add. One syntax for both.

update() — Merge Another Dict

# Merge another dict into the current one (modifies in place)
student = {"name": "Naveen", "age": 30}
additional = {"city": "Toronto", "age": 31}  # Note: "age" exists in both

student.update(additional)
print(student)  # {'name': 'Naveen', 'age': 31, 'city': 'Toronto'}
# "age" was overwritten by the incoming dict's value (31 replaces 30)

# update() also accepts keyword arguments
student.update(country="Canada", email="n@email.com")
print(student)  # {..., 'country': 'Canada', 'email': 'n@email.com'}

Removing Keys

student = {"name": "Naveen", "age": 30, "city": "Toronto", "email": "n@e.com"}

# del — remove by key (KeyError if missing)
del student["email"]
# del student["phone"]   # KeyError: 'phone'

# pop() — remove and RETURN the value (with optional default)
age = student.pop("age")
print(age)        # 30
print(student)    # {'name': 'Naveen', 'city': 'Toronto'}

# Safe pop with default (no error if missing)
phone = student.pop("phone", "N/A")
print(phone)      # "N/A" (key was missing, returned default)

# popitem() — remove and return the LAST inserted key-value pair
last = student.popitem()
print(last)       # ('city', 'Toronto')
print(student)    # {'name': 'Naveen'}

# clear() — remove all key-value pairs
student.clear()
print(student)    # {}

setdefault() — Get or Set

# setdefault: if key exists → return its value (do nothing)
#             if key missing → set it to the default AND return the default

student = {"name": "Naveen"}

# Key exists — returns existing value, does NOT overwrite
result = student.setdefault("name", "Unknown")
print(result)     # "Naveen" (existing value returned)
print(student)    # {'name': 'Naveen'} (unchanged)

# Key missing — sets it and returns the default
result = student.setdefault("city", "Toronto")
print(result)     # "Toronto" (new default returned)
print(student)    # {'name': 'Naveen', 'city': 'Toronto'} (key added!)

# Practical: grouping without defaultdict
groups = {}
items = [("fruit", "apple"), ("veg", "carrot"), ("fruit", "banana"), ("veg", "spinach")]
for category, item in items:
    groups.setdefault(category, []).append(item)
print(groups)  # {'fruit': ['apple', 'banana'], 'veg': ['carrot', 'spinach']}

Iterating Over Dictionaries

keys(), values(), items()

student = {"name": "Naveen", "age": 30, "city": "Toronto"}

# Iterate over KEYS (default behavior)
for key in student:
    print(key)          # name, age, city

# Explicitly iterate over keys
for key in student.keys():
    print(key)          # name, age, city

# Iterate over VALUES
for value in student.values():
    print(value)        # Naveen, 30, Toronto

# Iterate over KEY-VALUE PAIRS (most common and most useful)
for key, value in student.items():
    print(f"{key}: {value}")
# name: Naveen
# age: 30
# city: Toronto

# Convert to lists if needed
all_keys = list(student.keys())       # ['name', 'age', 'city']
all_values = list(student.values())   # ['Naveen', 30, 'Toronto']
all_items = list(student.items())     # [('name', 'Naveen'), ('age', 30), ...]

Iteration Patterns for Data Engineering

# Pattern 1: Find keys matching a condition
config = {"db_host": "server.com", "db_port": 1433, "api_key": "secret", "api_url": "https://..."}
db_settings = {k: v for k, v in config.items() if k.startswith("db_")}
print(db_settings)  # {'db_host': 'server.com', 'db_port': 1433}

# Pattern 2: Count values by category
orders = [{"status": "completed"}, {"status": "pending"}, {"status": "completed"},
          {"status": "failed"}, {"status": "completed"}]
counts = {}
for order in orders:
    status = order["status"]
    counts[status] = counts.get(status, 0) + 1
print(counts)  # {'completed': 3, 'pending': 1, 'failed': 1}

# Pattern 3: Build a lookup table from a list of dicts
employees = [
    {"id": 101, "name": "Alice", "dept": "Engineering"},
    {"id": 102, "name": "Bob", "dept": "Marketing"},
    {"id": 103, "name": "Charlie", "dept": "Engineering"},
]
lookup = {emp["id"]: emp for emp in employees}
print(lookup[102])  # {'id': 102, 'name': 'Bob', 'dept': 'Marketing'} — O(1) access!

Unpacking Dicts in Loops

# Iterate with enumerate (when you need an index)
for i, (key, value) in enumerate(student.items()):
    print(f"{i}: {key} = {value}")
# 0: name = Naveen
# 1: age = 30
# 2: city = Toronto

# Iterate two dicts in parallel with zip
names = {"en": "Hello", "es": "Hola", "fr": "Bonjour"}
codes = {"en": "English", "es": "Spanish", "fr": "French"}
for key in names:
    print(f"{codes[key]}: {names[key]}")
# English: Hello
# Spanish: Hola
# French: Bonjour

Dictionary Comprehensions

Basic Comprehension

# Syntax: {key_expr: value_expr for item in iterable}

# Create a dict from a list
names = ["alice", "bob", "charlie"]
name_lengths = {name: len(name) for name in names}
print(name_lengths)  # {'alice': 5, 'bob': 3, 'charlie': 7}

# Squares
squares = {n: n**2 for n in range(1, 6)}
print(squares)  # {1: 1, 2: 4, 3: 9, 4: 16, 5: 25}

# From two lists using zip
keys = ["name", "age", "city"]
values = ["Naveen", 30, "Toronto"]
person = {k: v for k, v in zip(keys, values)}
print(person)  # {'name': 'Naveen', 'age': 30, 'city': 'Toronto'}

Comprehension with Condition

# Filter: only even squares
even_squares = {n: n**2 for n in range(10) if n % 2 == 0}
print(even_squares)  # {0: 0, 2: 4, 4: 16, 6: 36, 8: 64}

# Filter dict: keep only non-null values
data = {"name": "Naveen", "email": None, "city": "Toronto", "phone": None}
clean = {k: v for k, v in data.items() if v is not None}
print(clean)  # {'name': 'Naveen', 'city': 'Toronto'}

# Transform values: uppercase all string values
config = {"host": "server.com", "port": 1433, "db": "mydb"}
upper_config = {k: v.upper() if isinstance(v, str) else v for k, v in config.items()}
print(upper_config)  # {'host': 'SERVER.COM', 'port': 1433, 'db': 'MYDB'}

Inverting a Dictionary

# Swap keys and values
country_codes = {"US": "United States", "CA": "Canada", "IN": "India"}
code_lookup = {v: k for k, v in country_codes.items()}
print(code_lookup)  # {'United States': 'US', 'Canada': 'CA', 'India': 'IN'}
print(code_lookup["Canada"])  # "CA"

# Warning: if values are not unique, later entries overwrite earlier ones
grades = {"Alice": "A", "Bob": "B", "Charlie": "A"}
inverted = {v: k for k, v in grades.items()}
print(inverted)  # {'A': 'Charlie', 'B': 'Bob'} — Alice is lost! (Charlie overwrote)

Transforming Keys or Values

# Rename keys
old_data = {"firstName": "Naveen", "lastName": "Vuppula", "phoneNumber": "416-555-1234"}
key_map = {"firstName": "first_name", "lastName": "last_name", "phoneNumber": "phone"}
new_data = {key_map.get(k, k): v for k, v in old_data.items()}
print(new_data)  # {'first_name': 'Naveen', 'last_name': 'Vuppula', 'phone': '416-555-1234'}

# Convert all keys to lowercase
messy = {"Name": "Naveen", "AGE": 30, "City": "Toronto"}
clean = {k.lower(): v for k, v in messy.items()}
print(clean)  # {'name': 'Naveen', 'age': 30, 'city': 'Toronto'}

Nested Dictionaries

Creating and Accessing Nested Dicts

# Dicts can contain other dicts — any level of nesting
company = {
    "name": "DriveDataScience",
    "location": {
        "city": "Toronto",
        "province": "Ontario",
        "country": "Canada"
    },
    "team": {
        "engineering": {
            "lead": "Naveen",
            "count": 5,
            "skills": ["Python", "SQL", "Azure", "Fabric"]
        },
        "marketing": {
            "lead": "Alice",
            "count": 3
        }
    }
}

# Access nested values with chained brackets
print(company["location"]["city"])                    # "Toronto"
print(company["team"]["engineering"]["lead"])          # "Naveen"
print(company["team"]["engineering"]["skills"][0])     # "Python" (list inside dict)

# This is exactly how JSON data looks — and why dicts are essential for API work

Real-life analogy: A nested dict is like a filing cabinet (outer dict) with drawers (first level), folders inside drawers (second level), and documents inside folders (third level). cabinet["drawer3"]["folder2"]["document1"] walks you to the exact document.

Safe Nested Access (Avoiding KeyError Chains)

# The problem: any missing key in the chain crashes everything
# company["team"]["data_science"]["lead"]  # KeyError: 'data_science'

# Solution 1: Chained get() with empty dict defaults
lead = company.get("team", {}).get("data_science", {}).get("lead", "Unknown")
print(lead)  # "Unknown" — no crash!

# Solution 2: try/except
try:
    lead = company["team"]["data_science"]["lead"]
except KeyError:
    lead = "Unknown"

# Solution 3: Write a helper function
def safe_get(d, *keys, default=None):
    """Safely navigate nested dicts."""
    for key in keys:
        if isinstance(d, dict):
            d = d.get(key)
        else:
            return default
        if d is None:
            return default
    return d

print(safe_get(company, "team", "engineering", "lead"))        # "Naveen"
print(safe_get(company, "team", "data_science", "lead"))       # None
print(safe_get(company, "team", "data_science", "lead", default="N/A"))  # "N/A"

Modifying Nested Dicts

# Add a new nested key
company["team"]["data_science"] = {"lead": "Bob", "count": 2}
print(company["team"]["data_science"]["lead"])  # "Bob"

# Update a deeply nested value
company["team"]["engineering"]["count"] = 6

# Be careful — if an intermediate key is missing, you cannot assign deep
# company["settings"]["theme"]["color"] = "blue"  # KeyError: 'settings'
# You must create each level first:
company["settings"] = {}
company["settings"]["theme"] = {}
company["settings"]["theme"]["color"] = "blue"

# Or use setdefault for safer construction:
company.setdefault("settings", {}).setdefault("theme", {})["color"] = "blue"

Flattening Nested Dicts

# Convert nested dict to flat dict with dot-separated keys
def flatten_dict(d, parent_key='', sep='.'):
    """Flatten a nested dict: {'a': {'b': 1}} → {'a.b': 1}"""
    items = []
    for k, v in d.items():
        new_key = f"{parent_key}{sep}{k}" if parent_key else k
        if isinstance(v, dict):
            items.extend(flatten_dict(v, new_key, sep).items())
        else:
            items.append((new_key, v))
    return dict(items)

nested = {"name": "Naveen", "location": {"city": "Toronto", "country": "Canada"}}
flat = flatten_dict(nested)
print(flat)  # {'name': 'Naveen', 'location.city': 'Toronto', 'location.country': 'Canada'}

# Useful for: converting nested JSON to flat table rows,
# logging structured data, creating Spark DataFrame columns

DefaultDict — Never Get a KeyError Again

defaultdict from the collections module automatically creates a default value for any missing key. No more checking “does this key exist?” before accessing it.

Real-life analogy: A regular dict is like a strict librarian — “we do not have that book” (KeyError). A defaultdict is like a helpful librarian — “we do not have that book, but I have created an empty shelf for it. Go ahead and put something there.”

defaultdict(int) — Counting

from collections import defaultdict

# Count word frequency
words = ["apple", "banana", "apple", "cherry", "banana", "apple"]

# WITHOUT defaultdict (verbose):
counts = {}
for word in words:
    if word in counts:
        counts[word] += 1
    else:
        counts[word] = 1
print(counts)  # {'apple': 3, 'banana': 2, 'cherry': 1}

# WITH defaultdict(int) (clean):
counts = defaultdict(int)   # Missing keys default to 0
for word in words:
    counts[word] += 1       # No KeyError — starts at 0
print(dict(counts))         # {'apple': 3, 'banana': 2, 'cherry': 1}

# How it works: int() returns 0 by default
# When you access counts["apple"] for the first time,
# defaultdict calls int() → creates counts["apple"] = 0 → then += 1

defaultdict(list) — Grouping

# Group items by category
items = [
    ("fruit", "apple"), ("veg", "carrot"), ("fruit", "banana"),
    ("dairy", "milk"), ("veg", "spinach"), ("fruit", "cherry"),
    ("dairy", "cheese")
]

# WITHOUT defaultdict:
groups = {}
for category, item in items:
    if category not in groups:
        groups[category] = []
    groups[category].append(item)

# WITH defaultdict(list):
groups = defaultdict(list)   # Missing keys default to []
for category, item in items:
    groups[category].append(item)   # No KeyError — starts with empty list

print(dict(groups))
# {'fruit': ['apple', 'banana', 'cherry'],
#  'veg': ['carrot', 'spinach'],
#  'dairy': ['milk', 'cheese']}

# Practical: group employees by department
employees = [("Eng", "Alice"), ("Mkt", "Bob"), ("Eng", "Charlie"), ("Mkt", "Diana")]
by_dept = defaultdict(list)
for dept, name in employees:
    by_dept[dept].append(name)
print(dict(by_dept))  # {'Eng': ['Alice', 'Charlie'], 'Mkt': ['Bob', 'Diana']}

defaultdict(set) — Unique Grouping

# Group with automatic deduplication
tags = [("post1", "python"), ("post1", "data"), ("post2", "python"),
        ("post1", "python"), ("post2", "sql")]  # Note: post1+python appears twice

tag_map = defaultdict(set)  # set = unique values
for post, tag in tags:
    tag_map[post].add(tag)

print(dict(tag_map))
# {'post1': {'python', 'data'}, 'post2': {'python', 'sql'}}
# Duplicate "python" for post1 was automatically ignored (set behavior)

defaultdict vs setdefault

Feature	defaultdict	setdefault
Import needed?	Yes (from collections)	No (built-in dict method)
When default is created	On any missing key access	Only when you call setdefault()
Readability	Very clean for loops	Slightly verbose
Works with	Separate defaultdict object	Any regular dict
Use when	Building a dict from scratch	Adding defaults to existing dict

Counter — Frequency Analysis

Creating a Counter

from collections import Counter

# Count anything iterable
word_counts = Counter(["apple", "banana", "apple", "cherry", "apple", "banana"])
print(word_counts)  # Counter({'apple': 3, 'banana': 2, 'cherry': 1})

# Count characters in a string
char_counts = Counter("mississippi")
print(char_counts)  # Counter({'s': 4, 'i': 4, 'p': 2, 'm': 1})

# Count from a dict
c = Counter({"a": 5, "b": 3, "c": 1})

# Access counts (missing keys return 0, not KeyError!)
print(word_counts["apple"])    # 3
print(word_counts["grape"])    # 0 (not KeyError!)

most_common()

# Get the N most common elements
print(word_counts.most_common(2))    # [('apple', 3), ('banana', 2)]
print(word_counts.most_common())     # All elements sorted by count

# Practical: find the most common error type in logs
errors = ["timeout", "auth_fail", "timeout", "disk_full", "timeout",
          "auth_fail", "timeout", "disk_full", "auth_fail"]
error_counts = Counter(errors)
print(error_counts.most_common(1))   # [('timeout', 4)] — most frequent error

Counter Arithmetic

# Counters support arithmetic operations
today = Counter({"apple": 10, "banana": 5, "cherry": 3})
yesterday = Counter({"apple": 8, "banana": 7, "cherry": 3})

# What changed?
increase = today - yesterday
print(increase)      # Counter({'apple': 2}) — apple increased by 2

decrease = yesterday - today
print(decrease)      # Counter({'banana': 2}) — banana decreased by 2

# Combined totals
total = today + yesterday
print(total)         # Counter({'apple': 18, 'banana': 12, 'cherry': 6})

OrderedDict and ChainMap

from collections import OrderedDict, ChainMap

# OrderedDict — guaranteed insertion order (less needed since Python 3.7+)
# Regular dicts now maintain insertion order, but OrderedDict still has:
# - move_to_end() method
# - Equality comparison considers order (regular dicts do not)

od = OrderedDict()
od["first"] = 1
od["second"] = 2
od.move_to_end("first")  # Move to end
print(list(od.keys()))    # ['second', 'first']

# ChainMap — search multiple dicts as one (useful for config layering)
defaults = {"color": "blue", "font": "Arial", "size": 12}
user_settings = {"color": "red"}
env_settings = {"font": "Helvetica"}

config = ChainMap(env_settings, user_settings, defaults)
print(config["color"])   # "red" (found in user_settings)
print(config["font"])    # "Helvetica" (found in env_settings)
print(config["size"])    # 12 (found in defaults)
# Searches first dict, then second, then third — first match wins

Merging Dictionaries

update() Method

# Modifies the original dict
a = {"x": 1, "y": 2}
b = {"y": 3, "z": 4}
a.update(b)
print(a)  # {'x': 1, 'y': 3, 'z': 4} — a is modified, b's "y" wins

Unpacking with **

# Creates a NEW dict (original unchanged)
a = {"x": 1, "y": 2}
b = {"y": 3, "z": 4}
merged = {**a, **b}
print(merged)  # {'x': 1, 'y': 3, 'z': 4} — b's "y" wins (last one wins)
print(a)       # {'x': 1, 'y': 2} — unchanged

# Merge with additional values
result = {**a, **b, "w": 5}
print(result)  # {'x': 1, 'y': 3, 'z': 4, 'w': 5}

Merge Operator | (Python 3.9+)

# | creates a new merged dict
a = {"x": 1, "y": 2}
b = {"y": 3, "z": 4}
merged = a | b
print(merged)  # {'x': 1, 'y': 3, 'z': 4}

# |= updates in place (like update)
a |= b
print(a)       # {'x': 1, 'y': 3, 'z': 4}

Which Values Win in a Merge

When the same key exists in both dicts, the LAST dict wins:

  a = {"x": 1, "y": 2}
  b = {"y": 3, "z": 4}

  {**a, **b}  → "y" comes from b (value 3) — b is unpacked LAST
  {**b, **a}  → "y" comes from a (value 2) — a is unpacked LAST

  a.update(b) → "y" comes from b (incoming dict wins)
  b.update(a) → "y" comes from a (incoming dict wins)

  Rule: the LAST writer wins. Like layers of paint — the top layer covers what is below.

Dictionaries as Function Arguments

**kwargs

# **kwargs collects keyword arguments into a dict
def create_user(**kwargs):
    print(type(kwargs))   # <class 'dict'>
    for key, value in kwargs.items():
        print(f"  {key}: {value}")

create_user(name="Naveen", age=30, city="Toronto")
# <class 'dict'>
#   name: Naveen
#   age: 30
#   city: Toronto

# Combine with regular and *args
def log_event(event_type, *tags, **details):
    print(f"Event: {event_type}")
    print(f"Tags: {tags}")
    print(f"Details: {details}")

log_event("login", "web", "mobile", user="Naveen", ip="1.2.3.4")
# Event: login
# Tags: ('web', 'mobile')
# Details: {'user': 'Naveen', 'ip': '1.2.3.4'}

Unpacking Dicts into Function Calls

# ** unpacks a dict into keyword arguments
def send_email(to, subject, body):
    print(f"To: {to}, Subject: {subject}")

email_data = {"to": "naveen@email.com", "subject": "Hello", "body": "Hi there!"}
send_email(**email_data)  # Same as: send_email(to=..., subject=..., body=...)

# Practical: pass config dict to database connection
config = {"host": "server.com", "port": 1433, "database": "mydb"}
# connection = connect(**config)  # Unpacks into keyword arguments

Real-World Data Engineering Patterns

Config Files and Settings

# Dicts are the natural structure for configuration
pipeline_config = {
    "source": {
        "type": "jdbc",
        "host": "source-server.database.windows.net",
        "database": "source_db",
        "tables": ["customers", "orders", "products"]
    },
    "sink": {
        "type": "delta",
        "path": "abfss://bronze@lake.dfs.core.windows.net/",
        "format": "parquet"
    },
    "settings": {
        "batch_size": 10000,
        "parallel": True,
        "retry_count": 3
    }
}

# Access config cleanly
for table in pipeline_config["source"]["tables"]:
    print(f"Processing {table}...")
    # process_table(table, pipeline_config["sink"]["path"])

JSON to Dict and Back

import json

# JSON string → Python dict
json_str = '{"name": "Naveen", "age": 30, "scores": [90, 85, 92]}'
data = json.loads(json_str)
print(type(data))         # <class 'dict'>
print(data["scores"][0])  # 90

# Python dict → JSON string
person = {"name": "Naveen", "city": "Toronto", "active": True}
json_output = json.dumps(person, indent=2)
print(json_output)
# {
#   "name": "Naveen",
#   "city": "Toronto",
#   "active": true
# }

# Read JSON file → dict
with open("config.json", "r") as f:
    config = json.load(f)

# Write dict → JSON file
with open("output.json", "w") as f:
    json.dump(data, f, indent=2)

Building Lookup Tables

# Convert a list of records to a fast lookup dict
# O(n) to build, then O(1) for every lookup

customers = [
    {"id": 1001, "name": "Alice", "city": "Toronto"},
    {"id": 1002, "name": "Bob", "city": "London"},
    {"id": 1003, "name": "Charlie", "city": "Delhi"},
]

# Build lookup by ID
customer_by_id = {c["id"]: c for c in customers}

# Now finding customer 1002 is instant — no scanning the list
print(customer_by_id[1002]["name"])  # "Bob" — O(1)

# Without lookup: must scan the list — O(n) every time
# for c in customers:
#     if c["id"] == 1002:
#         print(c["name"])   # O(n) — slow for large lists

Grouping and Aggregating Data

# Group orders by customer with defaultdict
from collections import defaultdict

orders = [
    {"customer": "Alice", "amount": 50}, {"customer": "Bob", "amount": 30},
    {"customer": "Alice", "amount": 70}, {"customer": "Charlie", "amount": 20},
    {"customer": "Bob", "amount": 45}, {"customer": "Alice", "amount": 25},
]

# Group: customer → list of amounts
by_customer = defaultdict(list)
for order in orders:
    by_customer[order["customer"]].append(order["amount"])
print(dict(by_customer))
# {'Alice': [50, 70, 25], 'Bob': [30, 45], 'Charlie': [20]}

# Aggregate: customer → total spend
totals = {customer: sum(amounts) for customer, amounts in by_customer.items()}
print(totals)  # {'Alice': 145, 'Bob': 75, 'Charlie': 20}

# Sort by total spend
top_customers = sorted(totals.items(), key=lambda x: x[1], reverse=True)
print(top_customers)  # [('Alice', 145), ('Bob', 75), ('Charlie', 20)]

Common Mistakes

1. Using [] instead of get() for possibly-missing keys — data["email"] crashes if "email" is missing. Use data.get("email", "N/A") for safe access. In data engineering, API responses and config files often have optional fields.
2. Modifying a dict while iterating over it — for key in my_dict: del my_dict[key] raises RuntimeError. Iterate over a copy: for key in list(my_dict.keys()): del my_dict[key].
3. Assuming dict order before Python 3.7 — dicts maintain insertion order in Python 3.7+. In earlier versions they do not. If you must support older Python, use OrderedDict.
4. Using mutable default values in dict.fromkeys() — dict.fromkeys(["a","b"], []) creates ONE shared list for all keys! Modifying one affects all. Use a dict comprehension instead: {k: [] for k in ["a","b"]}.
5. Not knowing that Counter returns 0 for missing keys — unlike regular dicts, Counter["missing_key"] returns 0, not KeyError. This is useful but can hide bugs if you expect a KeyError.
6. Inverting a dict with duplicate values — {v: k for k, v in d.items()} silently drops entries when multiple keys have the same value. The last key wins. Use defaultdict(list) for a safe inversion: inv = defaultdict(list); for k,v in d.items(): inv[v].append(k).
7. Deeply nesting without validation — data["a"]["b"]["c"]["d"] is fragile. Any missing key in the chain crashes. Use the safe_get() helper or chained .get() calls with default empty dicts.

Interview Questions

Q: How does a Python dictionary work internally? A: A dict is a hash table. When you set d["key"] = value, Python computes hash("key"), reduces it to a slot index via modulo, and stores the key-value pair in that slot. Lookup is O(1) because Python jumps directly to the slot instead of scanning. Collisions (two keys mapping to the same slot) are handled by probing.

Q: What is the difference between dict[key] and dict.get(key)? A: dict[key] raises KeyError if the key does not exist. dict.get(key, default) returns the default value instead (None if no default specified). Use [] when the key must exist (absence is a bug). Use get() when the key might be missing (absence is normal).

Q: What is defaultdict and when would you use it? A: defaultdict from collections automatically creates a default value for any missing key. defaultdict(int) defaults to 0 (for counting). defaultdict(list) defaults to [] (for grouping). It eliminates the need to check “if key in dict” before accessing or appending, making code cleaner and faster.

Q: How do you merge two dictionaries in Python? A: Three ways: (1) a.update(b) modifies a in place. (2) {**a, **b} creates a new merged dict. (3) a | b (Python 3.9+) creates a new merged dict. In all cases, when both dicts have the same key, the last dict’s value wins.

Q: When would you use Counter vs defaultdict(int)? A: Both can count, but Counter offers additional features: most_common(n) for top N items, arithmetic between Counters (add, subtract), and it returns 0 for missing keys instead of KeyError. Use Counter for frequency analysis and defaultdict(int) for general counting in loops.

Q: How do you safely access deeply nested dictionary keys? A: Chain .get() calls with empty dict defaults: data.get("a", {}).get("b", {}).get("c", "default"). If any level is missing, it returns the default instead of crashing. For complex nested structures, write a helper function that takes a dict and a list of keys, navigating one level at a time with get().

Q: Why can lists not be dictionary keys? A: Dict keys must be hashable (have a fixed hash value). Lists are mutable — if a list used as a key were modified, its hash would change, and the dict would lose track of which slot the key-value pair is in. Immutable types (str, int, tuple, frozenset) have stable hashes and can be used as keys. Convert a list to a tuple to use it as a key.

Wrapping Up

Dictionaries are Python’s most powerful data structure. They power namespaces, JSON, configs, lookup tables, and nearly every data transformation pipeline. Master the core operations (get vs brackets, update, comprehensions), the collections module (defaultdict for grouping, Counter for frequency), and nested dict patterns (safe access, flattening). These skills carry directly into data engineering where you work with JSON APIs, config-driven pipelines, and key-value transformations daily.

Previous in this series: Tuples, Sets, and Frozensets

Next in this series: Conditionals — if/elif/else, Ternary, Match-Case

---

Naveen Vuppula is a Senior Data Engineering Consultant and app developer based in Ontario, Canada. He writes about Python, SQL, AWS, Azure, and everything data engineering at DriveDataScience.com.