Python Regular Expressions: re Module, Pattern Matching, Groups, Lookaheads, Data Cleaning, Validation, and Every Pattern Data Engineers Need

A CSV arrives from a vendor. The phone numbers are in 15 different formats — (416) 555-0123, 416-555-0123, +1 416 555 0123, 4165550123. The emails have leading spaces, trailing commas, and some are clearly fake. The addresses mix street names, unit numbers, postal codes, and country codes into a single field. Your pipeline needs to clean, validate, and standardize all of this — and it changes every month because the vendor “updated their export format.”

This is what regular expressions solve. A regex is a pattern language that describes text structure. Instead of writing 15 if-statements for 15 phone formats, you write one pattern: \+?1?\s*\(?(\d{3})\)?[\s.-]*(\d{3})[\s.-]*(\d{4}). That single pattern matches all 15 formats and extracts the three number groups.

Real-life analogy: A regular expression is like a search warrant. It describes exactly what you are looking for — “a license plate with 4 letters followed by 3 digits” — without knowing the specific value. The police (the regex engine) checks every car (string) against the warrant (pattern). Matches are seized (extracted). Non-matches are ignored. The warrant can be broad (“any plate”) or surgical (“plate starting with ABCD, followed by 1-2-3, on a red car”). The more specific your warrant, the more precise your results.

Table of Contents

• Why Regular Expressions
• The re Module — Getting Started
• re.search() — Find the First Match
• re.match() — Match at the Start
• re.findall() — Find All Matches
• re.finditer() — Iterate Over Matches
• re.sub() — Search and Replace
• re.split() — Split on a Pattern
• re.compile() — Pre-Compile for Reuse
• Pattern Syntax — Building Blocks
• Literal Characters
• Character Classes ([ ])
• Predefined Character Classes (\d, \w, \s)
• Quantifiers (*, +, ?, {n})
• Greedy vs Lazy Quantifiers
• Anchors (^, $, \b)
• Alternation (|)
• Escaping Special Characters
• Groups and Capturing
• Basic Groups ( )
• Named Groups (?P<name>)
• Non-Capturing Groups (?:)
• Backreferences
• Lookahead and Lookbehind
• Positive Lookahead (?=)
• Negative Lookahead (?!)
• Positive Lookbehind (?<=)
• Negative Lookbehind (?<!)
• Flags
• re.IGNORECASE
• re.MULTILINE
• re.DOTALL
• re.VERBOSE
• Data Engineering Patterns
• Email Validation
• Phone Number Standardization
• Date Format Detection and Parsing
• IP Address Extraction
• Log File Parsing
• Column Name Cleaning
• PII Masking
• CSV Field Cleaning
• URL Parsing
• Regex Performance Tips
• Common Mistakes
• Interview Questions
• Wrapping Up

Why Regular Expressions

# WITHOUT regex — fragile, verbose, misses edge cases
def is_valid_email_manual(email):
    if "@" not in email:
        return False
    if "." not in email.split("@")[1]:
        return False
    if email.startswith("@"):
        return False
    if " " in email:
        return False
    # What about double dots? Leading dots? Special chars? 20+ more checks...
    return True

# WITH regex — one pattern captures the entire rule set
import re
def is_valid_email(email):
    pattern = r'^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}$'
    return bool(re.match(pattern, email))

# One line. Handles all edge cases. Readable once you know the syntax.

The re Module — Getting Started

re.search() — Find the First Match

import re

# re.search(pattern, string) — scans the ENTIRE string for a match
# Returns a Match object if found, None if not

text = "Customer ID: 12345, Order: 67890"

match = re.search(r'\d+', text)   # \d+ = one or more digits
if match:
    print(match.group())    # "12345" — the first match
    print(match.start())    # 13 — start position
    print(match.end())      # 18 — end position
    print(match.span())     # (13, 18) — start and end

# No match returns None — always check before calling .group()
match = re.search(r'xyz', text)
if match:
    print(match.group())
else:
    print("No match found")   # "No match found"

# IMPORTANT: Always use raw strings r'...' for patterns
# Without r: '\\d+' (have to escape the backslash)
# With r:    r'\d+' (backslash is literal — cleaner)

re.match() — Match at the Start

# re.match() only checks the BEGINNING of the string
# re.search() checks the ENTIRE string

text = "Error: Connection timeout at 14:30:00"

# match() — only matches if the pattern is at position 0
print(re.match(r'Error', text))     # <Match object> — "Error" is at the start
print(re.match(r'Connection', text)) # None — "Connection" is not at the start

# search() — finds it anywhere
print(re.search(r'Connection', text))  # <Match object> — found at position 7

# RULE: Use match() to validate that a string STARTS with a pattern
# Use search() to find a pattern ANYWHERE in the string
# For full-string validation, use match() with $ anchor: r'^pattern$'

re.findall() — Find All Matches

# findall() returns a LIST of all non-overlapping matches

text = "IDs: 12345, 67890, 11111, 99999"

# Find all numbers
numbers = re.findall(r'\d+', text)
print(numbers)   # ['12345', '67890', '11111', '99999']

# Find all email addresses in a document
doc = "Contact us at support@example.com or sales@company.org for help"
emails = re.findall(r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}', doc)
print(emails)   # ['support@example.com', 'sales@company.org']

# With groups — findall returns the GROUP content, not the full match
text = "Date: 2026-07-08, Next: 2026-07-15"
dates = re.findall(r'(\d{4})-(\d{2})-(\d{2})', text)
print(dates)   # [('2026', '07', '08'), ('2026', '07', '15')]
# Each match is a tuple of captured groups

Real-life analogy: re.search() is like a detective finding the first suspect matching a description. re.findall() is like rounding up every person matching the description. re.match() is like a bouncer who only checks the first person in line.

re.finditer() — Iterate Over Matches

# finditer() returns an ITERATOR of Match objects — more info than findall()
# Each match has .group(), .start(), .end(), .span(), and named groups

log = "2026-07-08 14:30:00 ERROR Connection failed\n2026-07-08 14:31:00 WARNING Retry 2/3\n2026-07-08 14:31:30 INFO Connected"

pattern = r'(\d{4}-\d{2}-\d{2}) (\d{2}:\d{2}:\d{2}) (\w+) (.+)'

for match in re.finditer(pattern, log):
    timestamp = match.group(1)
    time = match.group(2)
    level = match.group(3)
    message = match.group(4)
    print(f"[{level}] {timestamp} {time}: {message}")

# [ERROR] 2026-07-08 14:30:00: Connection failed
# [WARNING] 2026-07-08 14:31:00: Retry 2/3
# [INFO] 2026-07-08 14:31:30: Connected

# finditer is preferred over findall when you need:
# - Match positions (.start(), .end())
# - Named groups
# - Memory efficiency (lazy iteration vs loading all into a list)

re.sub() — Search and Replace

# re.sub(pattern, replacement, string) — replace all matches

# Remove all non-digit characters from a phone number
phone = "+1 (416) 555-0123"
clean = re.sub(r'[^\d]', '', phone)
print(clean)   # "14165550123"

# Replace multiple spaces with a single space
text = "Name:    Naveen     Vuppula"
clean = re.sub(r'\s+', ' ', text)
print(clean)   # "Name: Naveen Vuppula"

# Mask credit card numbers (keep last 4 digits)
text = "Card: 4111-1111-1111-1234"
masked = re.sub(r'\d{4}-\d{4}-\d{4}-(\d{4})', r'****-****-****-\1', text)
print(masked)   # "Card: ****-****-****-1234"
# \1 refers to the first captured group (the last 4 digits)

# Replace with a function — dynamic replacement
def upper_match(match):
    return match.group(0).upper()

text = "hello world python"
result = re.sub(r'\b\w', upper_match, text)   # Capitalize first letter of each word
print(result)   # "Hello World Python"

# Count replacements with re.subn()
result, count = re.subn(r'\d', 'X', "Order 123, Item 456")
print(f"Result: {result}, Replacements: {count}")
# Result: Order XXX, Item XXX, Replacements: 6

re.split() — Split on a Pattern

# Split a string using a regex pattern as the delimiter

# Split on any combination of commas, semicolons, and spaces
text = "apple, banana; cherry  date,fig"
parts = re.split(r'[,;\s]+', text)
print(parts)   # ['apple', 'banana', 'cherry', 'date', 'fig']

# Split on pipes (common in flat files)
record = "12345|Naveen|Toronto|Engineering"
fields = re.split(r'\|', record)
print(fields)   # ['12345', 'Naveen', 'Toronto', 'Engineering']

# Split but keep the delimiter
text = "Part1=value1&Part2=value2&Part3=value3"
parts = re.split(r'(&)', text)
print(parts)   # ['Part1=value1', '&', 'Part2=value2', '&', 'Part3=value3']

# Limit the number of splits
text = "error: file not found: /data/missing.csv"
parts = re.split(r':\s*', text, maxsplit=1)
print(parts)   # ['error', 'file not found: /data/missing.csv']

re.compile() — Pre-Compile for Reuse

# Compile a pattern once, use it many times — faster for repeated use

# Without compile — pattern is recompiled every iteration
for row in data:
    if re.match(r'^[A-Z]{2}\d{4}$', row["code"]):
        process(row)

# With compile — pattern is compiled ONCE
code_pattern = re.compile(r'^[A-Z]{2}\d{4}$')
for row in data:
    if code_pattern.match(row["code"]):
        process(row)

# Compiled patterns support the same methods:
email_pattern = re.compile(r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}')
email_pattern.search(text)
email_pattern.findall(text)
email_pattern.sub('***@***.***', text)
email_pattern.split(text)

# When to compile:
# - Pattern used in a loop (thousands+ iterations)
# - Pattern used across multiple functions
# - Complex patterns that benefit from a named variable
# When NOT to bother:
# - One-off usage (Python caches recent patterns internally)

Pattern Syntax — Building Blocks

Literal Characters

# Most characters match themselves literally
re.search(r'hello', 'say hello world')   # Matches "hello"
re.search(r'2026', 'year is 2026')       # Matches "2026"

# Special characters that need escaping: . ^ $ * + ? { } [ ] \ | ( )
# To match a literal dot: \.
# To match a literal dollar sign: \$
re.search(r'price: \$\d+\.\d{2}', 'price: $99.99')  # Matches "$99.99"

Character Classes ([ ])

# [abc]     — matches a, b, OR c (any ONE character in the set)
# [a-z]     — matches any lowercase letter
# [A-Z]     — matches any uppercase letter
# [0-9]     — matches any digit
# [a-zA-Z]  — matches any letter
# [^abc]    — matches anything EXCEPT a, b, or c (negation with ^)

re.findall(r'[aeiou]', 'hello world')       # ['e', 'o', 'o']
re.findall(r'[A-Z]', 'Hello World')          # ['H', 'W']
re.findall(r'[0-9]+', 'ID: 123, Age: 30')   # ['123', '30']
re.findall(r'[^0-9]', 'abc123')              # ['a', 'b', 'c']

# Practical: match hex color codes
re.findall(r'#[0-9a-fA-F]{6}', 'colors: #FF5733, #3498DB')
# ['#FF5733', '#3498DB']

Predefined Character Classes (\d, \w, \s)

Pattern	Matches	Equivalent	Opposite
\d	Any digit	[0-9]	\D (non-digit)
\w	Any word character	[a-zA-Z0-9_]	\W (non-word)
\s	Any whitespace	[ \t\n\r\f\v]	\S (non-whitespace)
.	Any character (except newline)	Almost everything	N/A

# \d — digits
re.findall(r'\d+', 'Order 12345, Item 67890')    # ['12345', '67890']

# \w — word characters (letters, digits, underscore)
re.findall(r'\w+', 'hello-world_test 123')        # ['hello', 'world_test', '123']

# \s — whitespace
re.split(r'\s+', 'hello   world\tnew\nline')      # ['hello', 'world', 'new', 'line']

# . — any character (except newline by default)
re.findall(r'c.t', 'cat cot cut c\nt')            # ['cat', 'cot', 'cut']

# Uppercase = OPPOSITE
re.findall(r'\D+', 'abc123def456')                 # ['abc', 'def'] (non-digits)
re.findall(r'\W+', 'hello, world!')                # [', ', '!'] (non-word chars)
re.findall(r'\S+', 'hello world')                  # ['hello', 'world'] (non-whitespace)

Quantifiers (*, +, ?, {n})

Quantifier	Meaning	Example Pattern	Matches
*	0 or more	ab*c	ac, abc, abbc, abbbc
+	1 or more	ab+c	abc, abbc, abbbc (NOT ac)
?	0 or 1 (optional)	colou?r	color, colour
{n}	Exactly n	\d{4}	1234 (exactly 4 digits)
{n,}	n or more	\d{2,}	12, 123, 1234 (2+ digits)
{n,m}	Between n and m	\d{2,4}	12, 123, 1234 (2-4 digits)

# Practical examples
re.findall(r'\d{3}-\d{3}-\d{4}', 'Call 416-555-0123 or 905-555-9876')
# ['416-555-0123', '905-555-9876']

re.findall(r'\b\w{5}\b', 'The quick brown fox jumps')
# ['quick', 'brown', 'jumps'] — exactly 5-letter words

# Optional prefix
re.findall(r'https?://\S+', 'Visit http://example.com or https://secure.com')
# ['http://example.com', 'https://secure.com'] — s is optional

Greedy vs Lazy Quantifiers

# GREEDY (default) — matches as MUCH as possible
# LAZY (add ?)      — matches as LITTLE as possible

html = '<b>bold</b> and <i>italic</i>'

# Greedy: .* grabs everything it can
print(re.findall(r'<.*>', html))
# ['<b>bold</b> and <i>italic</i>'] — ONE match, too much!

# Lazy: .*? grabs as little as possible
print(re.findall(r'<.*?>', html))
# ['<b>', '</b>', '<i>', '</i>'] — four separate tags, correct!

# The ? after a quantifier makes it lazy:
# *?  — 0 or more, as few as possible
# +?  — 1 or more, as few as possible
# ??  — 0 or 1, prefer 0
# {n,m}? — between n and m, prefer n

Real-life analogy: Greedy is like a hungry dog that eats everything in the bowl before stopping. Lazy is like a polite guest who takes one bite at a time and stops as soon as possible. When extracting HTML tags, you want the polite guest — grab each tag individually, not the entire page.

Anchors (^, $, \b)

# Anchors match POSITIONS, not characters

# ^  — start of string (or line with re.MULTILINE)
# $  — end of string (or line with re.MULTILINE)
# \b — word boundary (between \w and \W)

# ^ and $ for full-string validation
re.match(r'^\d{4}-\d{2}-\d{2}$', '2026-07-08')    # Match — entire string is a date
re.match(r'^\d{4}-\d{2}-\d{2}$', '2026-07-08 extra')  # None — extra text after date

# \b — word boundaries (prevents partial word matches)
text = "cat category caterpillar scattered"
re.findall(r'cat', text)       # ['cat', 'cat', 'cat', 'cat'] — matches inside words!
re.findall(r'\bcat\b', text)   # ['cat'] — only the standalone word "cat"

# Practical: match whole words only
re.findall(r'\bERROR\b', 'ERROR: ValueError not found')   # ['ERROR']
re.findall(r'\bERROR\b', 'ERRORCODE: 500')                # [] — no match (no boundary after ERROR)

Alternation (|)

# | means OR — match this pattern OR that pattern

# Match different log levels
re.findall(r'ERROR|WARNING|CRITICAL', log_text)

# Match different date separators
re.findall(r'\d{4}[-/\.]\d{2}[-/\.]\d{2}', 'dates: 2026-07-08, 2026/07/08, 2026.07.08')
# ['2026-07-08', '2026/07/08', '2026.07.08']

# Group alternation with parentheses
re.findall(r'(Mr|Mrs|Ms|Dr)\.\s\w+', 'Dr. Smith and Mrs. Jones')
# ['Dr', 'Mrs'] — captured groups

Escaping Special Characters

# Special regex characters: . ^ $ * + ? { } [ ] \ | ( )
# To match them literally, prefix with \

re.search(r'\$\d+\.\d{2}', 'Price: $99.99')        # Matches "$99.99"
re.search(r'file\.csv', 'open file.csv')             # Matches "file.csv"
re.search(r'\(416\)', 'Call (416) 555-0123')         # Matches "(416)"

# re.escape() — automatically escapes all special characters in a string
user_input = "price is $10.00 (USD)"
safe_pattern = re.escape(user_input)
print(safe_pattern)   # 'price\\ is\\ \\$10\\.00\\ \\(USD\\)'
# Use when building patterns from user input to prevent regex injection

Groups and Capturing

Basic Groups ( )

# Parentheses create GROUPS that capture matched text

# Extract parts of a date
match = re.search(r'(\d{4})-(\d{2})-(\d{2})', '2026-07-08')
if match:
    print(match.group(0))   # "2026-07-08" — full match
    print(match.group(1))   # "2026" — first group (year)
    print(match.group(2))   # "07" — second group (month)
    print(match.group(3))   # "08" — third group (day)
    print(match.groups())   # ('2026', '07', '08') — all groups as tuple

# Extract phone number parts
pattern = r'\(?(\d{3})\)?[\s.-]*(\d{3})[\s.-]*(\d{4})'
match = re.search(pattern, '(416) 555-0123')
if match:
    area, prefix, line = match.groups()
    standardized = f"{area}-{prefix}-{line}"
    print(standardized)   # "416-555-0123"

Named Groups (?P<name>)

# Named groups — access by name instead of number (much more readable)

pattern = r'(?P<year>\d{4})-(?P<month>\d{2})-(?P<day>\d{2})'
match = re.search(pattern, 'Date: 2026-07-08')

if match:
    print(match.group('year'))    # "2026"
    print(match.group('month'))   # "07"
    print(match.group('day'))     # "08"
    print(match.groupdict())      # {'year': '2026', 'month': '07', 'day': '08'}

# Named groups shine in complex patterns
log_pattern = re.compile(
    r'(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s+'
    r'(?P<level>DEBUG|INFO|WARNING|ERROR|CRITICAL)\s+'
    r'(?P<module>\S+)\s+'
    r'(?P<message>.+)'
)

line = "2026-07-08 14:30:00 ERROR pipeline.extract Connection timeout"
match = log_pattern.search(line)
if match:
    data = match.groupdict()
    print(data)
    # {'timestamp': '2026-07-08 14:30:00', 'level': 'ERROR',
    #  'module': 'pipeline.extract', 'message': 'Connection timeout'}

Non-Capturing Groups (?:)

# (?:...) groups without capturing — useful for applying quantifiers to a group
# without polluting match.groups()

# Capturing group — appears in results
re.findall(r'(https?)', 'http and https')   # ['http', 'https']

# Non-capturing group — NOT in results
re.findall(r'(?:https?://)\S+', 'visit http://example.com or https://secure.com')
# ['http://example.com', 'https://secure.com'] — full URLs, not just 'http'/'https'

# Practical: optional prefix without capturing
pattern = r'(?:USD\s?)?(\d+\.\d{2})'   # USD is optional, only capture the number
re.findall(pattern, 'Prices: USD 99.99, 45.00, USD 12.50')
# ['99.99', '45.00', '12.50'] — just the numbers

Backreferences

# \1, \2, etc. refer back to captured groups WITHIN the same pattern

# Find repeated words (common typo detection)
text = "The the quick brown fox fox"
dupes = re.findall(r'\b(\w+)\s+\1\b', text, re.IGNORECASE)
print(dupes)   # ['The', 'fox'] — words immediately repeated

# Find matching HTML tags
html = '<b>bold</b> <i>italic</i> <b>broken</i>'
matches = re.findall(r'<(\w+)>.*?</\1>', html)
print(matches)  # ['b', 'i'] — only properly closed tags

# In re.sub() — use \1 in replacement
text = "Last, First"
swapped = re.sub(r'(\w+), (\w+)', r'\2 \1', text)
print(swapped)  # "First Last"

Lookahead and Lookbehind

Lookaheads and lookbehinds assert that a pattern exists before or after the current position without consuming characters. They are “zero-width assertions” — they check but do not include the checked text in the match.

Real-life analogy: A lookahead is like saying “I want a house that is next to a park — but I am buying the house, not the park.” A lookbehind is like saying “I want a car that was previously owned by one person — the previous owner is the condition, not what I am buying.” The condition affects the match, but is not part of it.

# Positive lookahead (?=...)  — match only if FOLLOWED BY ...
# Match a number only if followed by " USD"
re.findall(r'\d+(?= USD)', 'Prices: 100 USD, 200 EUR, 300 USD')
# ['100', '300'] — only numbers followed by USD

# Negative lookahead (?!...)  — match only if NOT followed by ...
re.findall(r'\d+(?! USD)', 'Prices: 100 USD, 200 EUR, 300 USD')
# ['10', '200', '30'] — numbers NOT followed by USD (partial matches)

# Positive lookbehind (?<=...) — match only if PRECEDED BY ...
re.findall(r'(?<=\$)\d+', 'Prices: $100, €200, $300')
# ['100', '300'] — only numbers preceded by $

# Negative lookbehind (?<!...) — match only if NOT preceded by ...
re.findall(r'(?<!\$)\b\d+\b', 'IDs: 100, $200, 300')
# ['100', '300'] — numbers NOT preceded by $

# Practical: extract value after a label
text = "Name: Naveen, Age: 30, City: Toronto"
name = re.search(r'(?<=Name:\s)\w+', text).group()
print(name)   # "Naveen"

Flags

# re.IGNORECASE (re.I) — case-insensitive matching
re.findall(r'error', 'Error ERROR error', re.IGNORECASE)
# ['Error', 'ERROR', 'error']

# re.MULTILINE (re.M) — ^ and $ match start/end of EACH LINE
text = "Line 1\nLine 2\nLine 3"
re.findall(r'^Line', text)                    # ['Line'] — only start of string
re.findall(r'^Line', text, re.MULTILINE)      # ['Line', 'Line', 'Line'] — each line

# re.DOTALL (re.S) — dot (.) matches newlines too
text = "<div>\nHello\n</div>"
re.findall(r'<div>.*</div>', text)              # [] — . does not match \n
re.findall(r'<div>.*</div>', text, re.DOTALL)   # ['<div>\nHello\n</div>']

# re.VERBOSE (re.X) — allows comments and whitespace in patterns
# re.VERBOSE (re.X) — allows comments and whitespace in patterns
# Makes complex patterns readable by allowing line breaks and # comments
# Alternative: use string concatenation with inline comments (avoids escaping issues)
phone_pattern = re.compile(
    r'\+?1?'           # Optional country code (+1 or 1)
    r'[\s.-]*'         # Optional separator
    r'\(?(\d{3})\)?'   # Area code (with optional parens)
    r'[\s.-]*'         # Optional separator
    r'(\d{3})'         # Prefix
    r'[\s.-]*'         # Optional separator
    r'(\d{4})'         # Line number
)

# Combine flags with |
re.findall(r'^error.*$', text, re.IGNORECASE | re.MULTILINE)

Data Engineering Patterns

Email Validation

email_pattern = re.compile(
    r'^[a-zA-Z0-9._%+-]+'   # Local part
    r'@'                      # @ symbol
    r'[a-zA-Z0-9.-]+'        # Domain
    r'\.[a-zA-Z]{2,}$'       # TLD (.com, .org, .co.uk)
)

def validate_email(email):
    if not email or not isinstance(email, str):
        return False
    return bool(email_pattern.match(email.strip()))

# Test
print(validate_email("naveen@example.com"))     # True
print(validate_email("bad@.com"))                # False
print(validate_email("no-at-sign.com"))          # False
print(validate_email("  spaces@ok.com  "))       # True (stripped)

Phone Number Standardization

phone_pattern = re.compile(
    r'\+?1?'           # Optional country code
    r'[\s.(/-]*'       # Optional separators
    r'(\d{3})'         # Area code
    r'[\s.)\-/]*'      # Optional separators
    r'(\d{3})'         # Prefix
    r'[\s.\-/]*'       # Optional separators
    r'(\d{4})'         # Line number
)

def standardize_phone(phone):
    """Convert any phone format to (XXX) XXX-XXXX."""
    if not phone:
        return None
    match = phone_pattern.search(str(phone))
    if match:
        area, prefix, line = match.groups()
        return f"({area}) {prefix}-{line}"
    return None

# All of these produce the same output: "(416) 555-0123"
formats = [
    "(416) 555-0123",
    "416-555-0123",
    "416.555.0123",
    "+1 416 555 0123",
    "1-416-555-0123",
    "4165550123",
    "+1(416)555-0123",
]

for phone in formats:
    print(f"{phone:25s} → {standardize_phone(phone)}")

Date Format Detection and Parsing

# Detect which date format a string uses
DATE_PATTERNS = {
    "ISO":       re.compile(r'^\d{4}-\d{2}-\d{2}$'),
    "US":        re.compile(r'^\d{2}/\d{2}/\d{4}$'),
    "European":  re.compile(r'^\d{2}-\d{2}-\d{4}$'),
    "Long":      re.compile(r'^[A-Z][a-z]+ \d{1,2}, \d{4}$'),
    "Compact":   re.compile(r'^\d{8}$'),
}

def detect_date_format(date_str):
    for name, pattern in DATE_PATTERNS.items():
        if pattern.match(date_str.strip()):
            return name
    return "Unknown"

print(detect_date_format("2026-07-08"))      # "ISO"
print(detect_date_format("07/08/2026"))      # "US"
print(detect_date_format("08-07-2026"))      # "European"
print(detect_date_format("July 8, 2026"))    # "Long"
print(detect_date_format("20260708"))        # "Compact"

IP Address Extraction

ip_pattern = re.compile(
    r'\b(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})\b'
)

log = """
2026-07-08 14:30:00 Connection from 192.168.1.100 to 10.0.0.5
2026-07-08 14:30:05 Failed login from 203.0.113.42
2026-07-08 14:30:10 Request from 172.16.0.1 forwarded to 10.0.0.5
"""

ips = ip_pattern.findall(log)
print(ips)   # ['192.168.1.100', '10.0.0.5', '203.0.113.42', '172.16.0.1', '10.0.0.5']

# Unique IPs
unique_ips = sorted(set(ips))
print(unique_ips)
# ['10.0.0.5', '172.16.0.1', '192.168.1.100', '203.0.113.42']

Log File Parsing

# Parse structured log files into dictionaries
log_pattern = re.compile(
    r'(?P<timestamp>\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2})\s+'
    r'\|\s*(?P<level>\w+)\s*\|\s*'
    r'(?P<module>\S+)\s*\|\s*'
    r'(?P<message>.+)'
)

log_lines = [
    "2026-07-08 14:30:00 | INFO    | pipeline.extract | Starting extraction",
    "2026-07-08 14:30:05 | WARNING | pipeline.extract | Slow query: 8.2s",
    "2026-07-08 14:30:10 | ERROR   | pipeline.load    | Connection timeout",
]

for line in log_lines:
    match = log_pattern.search(line)
    if match:
        record = match.groupdict()
        print(record)

# {'timestamp': '2026-07-08 14:30:00', 'level': 'INFO', 'module': 'pipeline.extract', 'message': 'Starting extraction'}
# {'timestamp': '2026-07-08 14:30:05', 'level': 'WARNING', 'module': 'pipeline.extract', 'message': 'Slow query: 8.2s'}
# {'timestamp': '2026-07-08 14:30:10', 'level': 'ERROR', 'module': 'pipeline.load', 'message': 'Connection timeout'}

Column Name Cleaning

def clean_column_name(name):
    """Convert any column name to snake_case.

    'Customer First Name' → 'customer_first_name'
    'orderID'             → 'order_id'
    'Total (USD)'         → 'total_usd'
    'date-created'        → 'date_created'
    """
    # Insert underscore before uppercase letters (camelCase → camel_Case)
    name = re.sub(r'([a-z])([A-Z])', r'\1_\2', name)
    # Replace non-alphanumeric with underscore
    name = re.sub(r'[^a-zA-Z0-9]', '_', name)
    # Collapse multiple underscores
    name = re.sub(r'_+', '_', name)
    # Remove leading/trailing underscores and lowercase
    return name.strip('_').lower()

# Test with messy column names from different sources
columns = [
    "Customer First Name", "orderID", "Total (USD)",
    "date-created", "  Lead Source  ", "isActive",
    "CUSTOMER__ID", "phoneNumber", "Created At (UTC)",
]

for col in columns:
    print(f"  {col:25s} → {clean_column_name(col)}")
# Customer First Name       → customer_first_name
# orderID                   → order_id
# Total (USD)               → total_usd
# date-created              → date_created
#   Lead Source              → lead_source
# isActive                  → is_active
# CUSTOMER__ID              → customer_id
# phoneNumber               → phone_number
# Created At (UTC)          → created_at_utc

PII Masking

def mask_pii(text):
    """Mask personally identifiable information in text."""
    # Mask email addresses
    text = re.sub(
        r'[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}',
        '***@***.***',
        text
    )
    # Mask phone numbers (10+ digits with separators)
    text = re.sub(
        r'\+?1?[\s.-]*\(?\d{3}\)?[\s.-]*\d{3}[\s.-]*\d{4}',
        '***-***-****',
        text
    )
    # Mask SSN (XXX-XX-XXXX)
    text = re.sub(r'\d{3}-\d{2}-\d{4}', '***-**-****', text)

    # Mask credit card numbers (13-19 digits with separators)
    text = re.sub(r'\d{4}[\s-]?\d{4}[\s-]?\d{4}[\s-]?\d{4}', '****-****-****-****', text)

    return text

# Test
record = "Customer naveen@example.com, phone (416) 555-0123, SSN 123-45-6789, card 4111-1111-1111-1234"
print(mask_pii(record))
# "Customer ***@***.***,  phone ***-***-****, SSN ***-**-****, card ****-****-****-****"

CSV Field Cleaning

def clean_csv_field(value):
    """Clean a raw CSV field value."""
    if not isinstance(value, str):
        return value

    # Remove leading/trailing whitespace and quotes
    value = value.strip().strip('"').strip("'")

    # Replace multiple whitespace with single space
    value = re.sub(r'\s+', ' ', value)

    # Remove non-printable characters
    value = re.sub(r'[^\x20-\x7E]', '', value)

    # Remove trailing commas and semicolons
    value = re.sub(r'[,;]+$', '', value)

    return value.strip()

# Test with messy CSV data
messy = [
    '  "  Naveen   Vuppula  " ',
    "Toronto,\t  ON",
    "Hello\x00World\x01Test",
    "value1, value2,,,,",
]

for field in messy:
    print(f"  '{field}' → '{clean_csv_field(field)}'")

URL Parsing

url_pattern = re.compile(
    r'(?P<protocol>https?)://'      # Protocol
    r'(?P<domain>[^/:\s]+)'         # Domain
    r'(?::(?P<port>\d+))?'          # Optional port
    r'(?P<path>/[^\s?]*)?'          # Optional path
    r'(?:\?(?P<query>\S+))?'        # Optional query string
)

urls = [
    "https://api.example.com:8080/v2/customers?limit=100&offset=0",
    "http://localhost:5000/health",
    "https://drivedatascience.com/python-regex",
]

for url in urls:
    match = url_pattern.search(url)
    if match:
        print(match.groupdict())

Regex Performance Tips

1. Compile patterns used in loops — re.compile() once outside the loop. Python caches recent patterns, but compilation still has overhead.
2. Use raw strings — always r'...' for patterns. Avoids double-escaping and makes patterns readable.
3. Anchor when possible — ^ and $ let the engine fail fast instead of scanning the entire string.
4. Avoid .* when you can be specific — .* backtracks heavily. Use [^,]+ (everything except comma) or \S+ (non-whitespace) instead.
5. Prefer non-capturing groups — (?:...) is faster than (...) when you do not need the captured text.
6. Use finditer() over findall() for large texts — finditer() is lazy (processes one match at a time), findall() builds the entire list in memory.
7. Do not use regex for everything — simple string operations (str.startswith(), str.endswith(), str.replace(), in) are faster than regex for simple checks.

Common Mistakes

1. Forgetting raw strings — '\d+' is a string with \d, which Python may interpret as an escape sequence. Always use r'\d+'. The r prefix tells Python “do not interpret backslashes.”
2. Using re.match() when you mean re.search() — match() only checks the beginning of the string. If you want to find a pattern anywhere, use search(). This is the most common regex bug in Python.
3. Greedy matching by default — .* matches as much as possible. When extracting content between delimiters (tags, quotes), use .*? (lazy) to get individual matches instead of one giant match.
4. Not escaping special characters — . in regex matches ANY character, not just a literal dot. To match a literal dot (like in file.csv), use \.. Same for $, *, +, ?, (, ), etc.
5. Using .seconds in timedelta — wait, wrong post. But the equivalent regex mistake is confusing group(0) (full match) with group(1) (first capture group). group(0) is the entire match; group(1) is the first parenthesized group.
6. Not handling no-match cases — re.search() returns None when there is no match. Calling .group() on None raises AttributeError. Always check: if match: before accessing groups.
7. Over-using regex for simple operations — re.sub(r'hello', 'hi', text) is slower than text.replace('hello', 'hi'). Use string methods for literal replacements; reserve regex for patterns.

Interview Questions

Q: What is the difference between re.match() and re.search()? A: re.match() checks for a pattern only at the beginning of the string (position 0). re.search() scans the entire string for the first occurrence of the pattern anywhere. For full-string validation, use re.match() with a $ anchor: re.match(r'^\d{4}-\d{2}-\d{2}$', text). For finding a pattern inside a larger text, use re.search().

Q: What is the difference between greedy and lazy quantifiers? A: Greedy quantifiers (*, +) match as much text as possible. Lazy quantifiers (*?, +?) match as little as possible. Example: for <b>bold</b>, the greedy pattern <.*> matches the entire string <b>bold</b>. The lazy pattern <.*?> matches each tag separately: <b> and </b>.

Q: What are capturing groups and named groups? A: Capturing groups (pattern) extract matched text that you can access with match.group(1), group(2), etc. Named groups (?P<name>pattern) do the same but let you access by name: match.group('name') or match.groupdict(). Named groups make complex patterns self-documenting and easier to maintain.

Q: What is a lookahead and when would you use one? A: A lookahead (?=...) asserts that what follows the current position matches the given pattern, without consuming characters. Example: \d+(?= USD) matches numbers only if followed by ” USD” — but ” USD” is not part of the match. Use lookaheads for conditional matching where the context determines validity but should not be extracted.

Q: How do you clean column names from different source systems? A: Use a three-step regex approach: (1) Insert underscores before uppercase letters to handle camelCase: re.sub(r'([a-z])([A-Z])', r'\1_\2', name). (2) Replace all non-alphanumeric characters with underscores: re.sub(r'[^a-zA-Z0-9]', '_', name). (3) Collapse multiple underscores and lowercase: re.sub(r'_+', '_', name).strip('_').lower(). This converts “Customer First Name”, “orderID”, and “Total (USD)” all to consistent snake_case.

Q: When should you NOT use regex? A: Do not use regex for simple string operations — text.startswith('prefix') is faster than re.match(r'^prefix', text), and text.replace('old', 'new') is faster than re.sub(r'old', 'new', text). Do not use regex to parse complex structured formats like JSON, XML, or HTML — use a proper parser (json module, BeautifulSoup, lxml). And do not use regex for context-free grammars (matching nested parentheses at arbitrary depth), which regex cannot handle.

Wrapping Up

Regular expressions are the Swiss Army knife of text processing. The re module gives you search() for finding, findall() for extracting, sub() for replacing, split() for splitting, and compile() for reuse. The pattern syntax — character classes, quantifiers, anchors, groups, and lookaheads — lets you describe any text pattern precisely.

For data engineering, the key patterns are: email validation, phone number standardization, date format detection, log file parsing, column name cleaning, PII masking, and CSV field cleaning. These patterns appear in every ingestion pipeline that handles real-world data from external systems. Master the building blocks, and you can construct any pattern you need.

Previous in this series: Dates, Times, Timezones — datetime, timedelta, and Scheduling

Next in this series: Decorators and Context Managers

---

Naveen Vuppula is a Senior Data Engineering Consultant and app developer based in Ontario, Canada. He writes about Python, SQL, AWS, Azure, and everything data engineering at DriveDataScience.com.