Python Dates, Times, and Timezones: datetime, timedelta, strftime, strptime, zoneinfo, Scheduling, and Every Pattern Data Engineers Need

Every data pipeline cares about time. When was this file last modified? Which rows changed since yesterday? What timezone is the source system in? Should the watermark be midnight UTC or midnight EST? How do you partition data by year/month/day? How do you calculate “3 business days from now”? How do you parse the timestamp “07-08-2026 02:30 PM IST” into something your pipeline can compare and filter?

Dates and times look simple until they are not. Daylight saving time changes your UTC offset twice a year. Naive datetimes (no timezone) silently compare with timezone-aware datetimes — and produce wrong results. strftime and strptime look identical but do opposite things. And every source system seems to invent its own date format.

Real-life analogy: Think of dates and times like international flights. A flight departing Toronto at 8:00 AM lands in London at 8:00 PM — but that is not a 12-hour flight. Toronto is UTC-5 (or UTC-4 during DST) and London is UTC+0 (or UTC+1 during BST). The actual flight is about 7 hours. If you ignore timezones and just subtract the numbers, you get 12 hours — wrong. A naive datetime is like a clock with no timezone label — you see “8:00” but you do not know if it is Toronto 8:00 or London 8:00. An aware datetime is like a clock that says “8:00 AM EST” — now you know exactly what moment in time it represents, and you can compare it with any other clock in the world.

Table of Contents

• The datetime Module — Core Classes
• date — Dates Without Time
• time — Times Without Dates
• datetime — Dates and Times Together
• Creating datetimes
• Getting the Current Date and Time
• Accessing Components
• timedelta — Date and Time Math
• Adding and Subtracting Time
• Calculating Differences
• Practical timedelta Operations
• Formatting Dates — strftime (datetime → string)
• Common Format Codes
• Formatting for File Names and Paths
• Formatting for Display and Reports
• Parsing Dates — strptime (string → datetime)
• Parsing Common Formats
• Handling Multiple Date Formats
• Parsing with Error Handling
• The strftime vs strptime Memory Trick
• Timezones
• Naive vs Aware Datetimes
• zoneinfo (Python 3.9+)
• Converting Between Timezones
• UTC — The Universal Standard
• Why You Should Always Store UTC
• Daylight Saving Time Pitfalls
• ISO 8601 — The Universal Date Format
• What ISO 8601 Is
• Parsing and Formatting ISO 8601
• Working with Timestamps
• Unix Timestamps (Epoch Time)
• Converting Between datetime and Timestamps
• Data Engineering Patterns
• Date-Partitioned File Paths
• Watermark Pattern for Incremental Loads
• Pipeline Scheduling Windows
• Audit Timestamps
• Date Range Generator
• Business Days Calculator
• Common Mistakes
• Interview Questions
• Wrapping Up

The datetime Module — Core Classes

Python’s datetime module has four core classes. Understanding which one to use is the first step.

Class	What It Represents	Example
date	A calendar date (year, month, day)	2026-07-08
time	A time of day (hour, minute, second)	14:30:00
datetime	A date AND time combined	2026-07-08 14:30:00
timedelta	A duration (difference between two datetimes)	3 days, 5 hours, 30 minutes

date — Dates Without Time

from datetime import date

# Create a specific date
birthday = date(1995, 3, 15)        # March 15, 1995
print(birthday)                      # 1995-03-15

# Today's date
today = date.today()                 # 2026-07-08
print(today.year)                    # 2026
print(today.month)                   # 7
print(today.day)                     # 8

# Day of the week (Monday=0, Sunday=6)
print(today.weekday())               # 1 (Tuesday)
print(today.isoweekday())            # 2 (Tuesday — ISO: Monday=1, Sunday=7)

# Create from ISO format string
d = date.fromisoformat("2026-07-08") # Same as date(2026, 7, 8)

# Compare dates
yesterday = date(2026, 7, 7)
print(today > yesterday)             # True
print(today == date(2026, 7, 8))     # True

time — Times Without Dates

from datetime import time

# Create a specific time
morning = time(8, 30, 0)             # 08:30:00
print(morning.hour)                  # 8
print(morning.minute)                # 30
print(morning.second)                # 0

# With microseconds
precise = time(14, 30, 15, 123456)   # 14:30:15.123456
print(precise.microsecond)           # 123456

# time objects are rarely used alone in data engineering
# datetime (date + time) is almost always what you need

datetime — Dates and Times Together

Creating datetimes

from datetime import datetime

# Create a specific datetime
dt = datetime(2026, 7, 8, 14, 30, 0)   # July 8, 2026 at 2:30 PM
print(dt)                                # 2026-07-08 14:30:00

# From ISO format string
dt = datetime.fromisoformat("2026-07-08T14:30:00")
print(dt)                                # 2026-07-08 14:30:00

# From a date object (adds midnight time)
from datetime import date
d = date(2026, 7, 8)
dt = datetime.combine(d, time.min)       # 2026-07-08 00:00:00

# Replace specific components
dt = datetime(2026, 7, 8, 14, 30)
midnight = dt.replace(hour=0, minute=0, second=0)   # 2026-07-08 00:00:00
end_of_day = dt.replace(hour=23, minute=59, second=59)  # 2026-07-08 23:59:59

Getting the Current Date and Time

from datetime import datetime, timezone

# Current local datetime (NAIVE — no timezone info)
now = datetime.now()
print(now)   # 2026-07-08 14:30:00.123456

# Current UTC datetime (AWARE — has timezone info)
utc_now = datetime.now(timezone.utc)
print(utc_now)   # 2026-07-08 18:30:00.123456+00:00

# RECOMMENDATION: Always use datetime.now(timezone.utc) for timestamps
# in data pipelines. "now()" without timezone is ambiguous —
# it returns the server's local time, which changes if you deploy
# to a different region or if DST kicks in.

Accessing Components

dt = datetime(2026, 7, 8, 14, 30, 45)

print(dt.year)        # 2026
print(dt.month)       # 7
print(dt.day)         # 8
print(dt.hour)        # 14
print(dt.minute)      # 30
print(dt.second)      # 45
print(dt.date())      # date(2026, 7, 8) — extract just the date
print(dt.time())      # time(14, 30, 45) — extract just the time
print(dt.weekday())   # 1 (Tuesday)
print(dt.timestamp()) # 1783706245.0 — Unix timestamp (seconds since 1970-01-01)

timedelta — Date and Time Math

Adding and Subtracting Time

from datetime import datetime, timedelta

now = datetime(2026, 7, 8, 14, 30, 0)

# Add time
tomorrow = now + timedelta(days=1)           # 2026-07-09 14:30:00
next_week = now + timedelta(weeks=1)         # 2026-07-15 14:30:00
two_hours_later = now + timedelta(hours=2)   # 2026-07-08 16:30:00
in_90_minutes = now + timedelta(minutes=90)  # 2026-07-08 16:00:00

# Subtract time
yesterday = now - timedelta(days=1)          # 2026-07-07 14:30:00
last_month = now - timedelta(days=30)        # 2026-06-08 14:30:00
one_hour_ago = now - timedelta(hours=1)      # 2026-07-08 13:30:00

# Combine multiple units
dt = now + timedelta(days=3, hours=5, minutes=30)  # 3 days, 5.5 hours from now

# Negative timedelta
dt = now + timedelta(days=-7)                # Same as now - timedelta(days=7)

Real-life analogy: timedelta is like a stopwatch or a timer. It does not represent a specific moment (“July 8 at 2:30 PM”) — it represents a duration (“3 days and 5 hours”). You can add it to any datetime to get a new datetime. “3 days from now” is now + timedelta(days=3). “1 week ago” is now - timedelta(weeks=1).

Calculating Differences

# Subtracting two datetimes gives you a timedelta
start = datetime(2026, 7, 1, 8, 0, 0)
end = datetime(2026, 7, 8, 14, 30, 45)

duration = end - start
print(duration)               # 7 days, 6:30:45
print(duration.days)          # 7
print(duration.seconds)       # 23445 (the seconds WITHIN the last day)
print(duration.total_seconds()) # 628245.0 (TOTAL seconds — this is what you usually want)

# Common pattern: measure how long something took
import time
start_time = datetime.now()
# ... run pipeline ...
time.sleep(2)   # Simulating work
end_time = datetime.now()
elapsed = (end_time - start_time).total_seconds()
print(f"Pipeline completed in {elapsed:.1f} seconds")  # Pipeline completed in 2.0 seconds

# WARNING: duration.seconds is NOT total seconds!
# It is only the seconds component of the LAST partial day
d = timedelta(days=2, hours=3, seconds=45)
print(d.seconds)          # 10845 (3 hours * 3600 + 45 seconds) — NOT total
print(d.total_seconds())  # 183645.0 (2 days * 86400 + 10845) — TOTAL

Practical timedelta Operations

# Format a duration as hours:minutes:seconds
def format_duration(td):
    """Convert a timedelta to a human-readable string."""
    total_seconds = int(td.total_seconds())
    hours, remainder = divmod(total_seconds, 3600)
    minutes, seconds = divmod(remainder, 60)
    if hours > 0:
        return f"{hours}h {minutes}m {seconds}s"
    elif minutes > 0:
        return f"{minutes}m {seconds}s"
    else:
        return f"{seconds}s"

elapsed = timedelta(hours=2, minutes=15, seconds=30)
print(format_duration(elapsed))   # "2h 15m 30s"

elapsed = timedelta(seconds=45)
print(format_duration(elapsed))   # "45s"

# Compare timedeltas
if elapsed > timedelta(minutes=5):
    print("Pipeline took longer than 5 minutes — investigate")

# Multiply / divide timedeltas
half = timedelta(hours=4) / 2     # timedelta(hours=2)
double = timedelta(hours=4) * 2   # timedelta(hours=8)

Formatting Dates — strftime (datetime → string)

strftime converts a datetime object into a formatted string. The name stands for string format time — it formats a datetime into a string.

Common Format Codes

Code	Meaning	Example
%Y	4-digit year	2026
%y	2-digit year	26
%m	Month (zero-padded)	07
%d	Day (zero-padded)	08
%H	Hour 24-hour (zero-padded)	14
%I	Hour 12-hour (zero-padded)	02
%M	Minute (zero-padded)	30
%S	Second (zero-padded)	45
%p	AM/PM	PM
%A	Full weekday name	Tuesday
%a	Abbreviated weekday	Tue
%B	Full month name	July
%b	Abbreviated month	Jul
%Z	Timezone name	UTC, EST, IST
%z	UTC offset	+0000, -0500

Formatting for File Names and Paths

from datetime import datetime

now = datetime(2026, 7, 8, 14, 30, 0)

# Date-partitioned folder paths (data lake standard)
folder = now.strftime("year=%Y/month=%m/day=%d")
print(folder)   # "year=2026/month=07/day=08"

# File names with timestamps
filename = now.strftime("customers_%Y%m%d_%H%M%S.parquet")
print(filename)  # "customers_20260708_143000.parquet"

# Log file names
log_name = now.strftime("pipeline_%Y-%m-%d.log")
print(log_name)  # "pipeline_2026-07-08.log"

# Partition values for Delta tables
year = now.strftime("%Y")    # "2026"
month = now.strftime("%m")   # "07"
day = now.strftime("%d")     # "08"

Formatting for Display and Reports

now = datetime(2026, 7, 8, 14, 30, 0)

# Human-readable
print(now.strftime("%B %d, %Y"))           # "July 08, 2026"
print(now.strftime("%A, %B %d, %Y"))       # "Wednesday, July 08, 2026"
print(now.strftime("%d-%b-%Y"))            # "08-Jul-2026"
print(now.strftime("%I:%M %p"))            # "02:30 PM"

# ISO 8601 (standard for APIs and data exchange)
print(now.strftime("%Y-%m-%dT%H:%M:%S"))   # "2026-07-08T14:30:00"

# Pipeline audit log format
print(now.strftime("%Y-%m-%d %H:%M:%S"))   # "2026-07-08 14:30:00"

Parsing Dates — strptime (string → datetime)

strptime converts a string into a datetime object. The name stands for string parse time — it parses a string into a datetime.

Parsing Common Formats

from datetime import datetime

# Parse ISO format
dt = datetime.strptime("2026-07-08", "%Y-%m-%d")
print(dt)   # 2026-07-08 00:00:00

# Parse with time
dt = datetime.strptime("2026-07-08 14:30:00", "%Y-%m-%d %H:%M:%S")
print(dt)   # 2026-07-08 14:30:00

# Parse US format (MM/DD/YYYY)
dt = datetime.strptime("07/08/2026", "%m/%d/%Y")
print(dt)   # 2026-07-08 00:00:00

# Parse European format (DD-MM-YYYY)
dt = datetime.strptime("08-07-2026", "%d-%m-%Y")
print(dt)   # 2026-07-08 00:00:00

# Parse with AM/PM
dt = datetime.strptime("07/08/2026 02:30 PM", "%m/%d/%Y %I:%M %p")
print(dt)   # 2026-07-08 14:30:00

# Parse abbreviated month
dt = datetime.strptime("08-Jul-2026", "%d-%b-%Y")
print(dt)   # 2026-07-08 00:00:00

# Parse full month name
dt = datetime.strptime("July 8, 2026", "%B %d, %Y")
print(dt)   # 2026-07-08 00:00:00

Handling Multiple Date Formats

# Source systems love sending dates in random formats
# Build a multi-format parser that tries each format

def parse_date_flexible(date_str):
    """Try multiple date formats and return the first match."""
    formats = [
        "%Y-%m-%d",              # 2026-07-08
        "%Y-%m-%dT%H:%M:%S",    # 2026-07-08T14:30:00
        "%Y-%m-%d %H:%M:%S",    # 2026-07-08 14:30:00
        "%m/%d/%Y",              # 07/08/2026
        "%d-%m-%Y",              # 08-07-2026
        "%d/%m/%Y",              # 08/07/2026
        "%d-%b-%Y",              # 08-Jul-2026
        "%B %d, %Y",             # July 8, 2026
        "%m/%d/%Y %I:%M %p",    # 07/08/2026 02:30 PM
    ]

    for fmt in formats:
        try:
            return datetime.strptime(date_str.strip(), fmt)
        except ValueError:
            continue

    raise ValueError(f"Cannot parse date: '{date_str}' — no matching format found")

# Usage
print(parse_date_flexible("2026-07-08"))           # 2026-07-08 00:00:00
print(parse_date_flexible("07/08/2026"))           # 2026-07-08 00:00:00
print(parse_date_flexible("08-Jul-2026"))          # 2026-07-08 00:00:00
print(parse_date_flexible("July 8, 2026"))         # 2026-07-08 00:00:00

Parsing with Error Handling

# Always wrap strptime in try/except — source data WILL have bad dates

def safe_parse_date(date_str, fmt="%Y-%m-%d"):
    """Parse a date string safely, return None on failure."""
    if not date_str or date_str.strip() == "":
        return None
    try:
        return datetime.strptime(date_str.strip(), fmt)
    except (ValueError, TypeError):
        return None

# Usage in a data pipeline
for row in data:
    row["parsed_date"] = safe_parse_date(row.get("created_at"))
    if row["parsed_date"] is None:
        quarantine.append(row)   # Bad date — quarantine for investigation

The strftime vs strptime Memory Trick

These two methods have nearly identical names and are easy to confuse. Here is the trick:

• strftime — the f stands for format. It formats a datetime into a string. datetime → string.
• strptime — the p stands for parse. It parses a string into a datetime. string → datetime.

Real-life analogy: strftime is like a printer — it takes a datetime and prints it as a string in whatever format you want. strptime is like a scanner — it reads a string and converts it back into a datetime. Printer = format (f). Scanner = parse (p).

Timezones

Naive vs Aware Datetimes

from datetime import datetime, timezone

# NAIVE datetime — no timezone info (dangerous for pipelines)
naive = datetime(2026, 7, 8, 14, 30, 0)
print(naive.tzinfo)   # None — Python does not know what timezone this is

# AWARE datetime — has timezone info (safe for pipelines)
aware = datetime(2026, 7, 8, 14, 30, 0, tzinfo=timezone.utc)
print(aware.tzinfo)   # UTC
print(aware)           # 2026-07-08 14:30:00+00:00

# WARNING: You CANNOT compare naive and aware datetimes
# naive < aware   → TypeError: can't compare offset-naive and offset-aware datetimes
# This is a GOOD thing — Python is telling you the comparison is meaningless

# RULE FOR PIPELINES: Always use aware datetimes with UTC
# Store in UTC, convert to local only for display

Real-life analogy: A naive datetime is like a price tag that says “$100” with no currency. Is that USD? CAD? EUR? You cannot compare it with a price tag that says “€80” because you do not know the exchange rate. An aware datetime is like a price tag that says “$100 USD” — now you can convert it to any other currency and compare accurately.

zoneinfo (Python 3.9+)

from datetime import datetime
from zoneinfo import ZoneInfo

# Create timezone-aware datetimes
toronto = datetime(2026, 7, 8, 14, 30, tzinfo=ZoneInfo("America/Toronto"))
london = datetime(2026, 7, 8, 19, 30, tzinfo=ZoneInfo("Europe/London"))
mumbai = datetime(2026, 7, 9, 0, 0, tzinfo=ZoneInfo("Asia/Kolkata"))

print(toronto)   # 2026-07-08 14:30:00-04:00  (EDT in summer)
print(london)    # 2026-07-08 19:30:00+01:00  (BST in summer)
print(mumbai)    # 2026-07-09 00:00:00+05:30  (IST, no DST)

# All three represent the SAME moment in time!
print(toronto == london == mumbai)   # True

# Common timezone names:
# America/Toronto, America/New_York, America/Chicago, America/Los_Angeles
# Europe/London, Europe/Paris, Europe/Berlin
# Asia/Kolkata (India), Asia/Tokyo, Asia/Shanghai
# UTC (universal)

Converting Between Timezones

from datetime import datetime
from zoneinfo import ZoneInfo

# Create a UTC datetime
utc_time = datetime(2026, 7, 8, 18, 30, 0, tzinfo=ZoneInfo("UTC"))

# Convert to other timezones
toronto_time = utc_time.astimezone(ZoneInfo("America/Toronto"))
london_time = utc_time.astimezone(ZoneInfo("Europe/London"))
mumbai_time = utc_time.astimezone(ZoneInfo("Asia/Kolkata"))

print(f"UTC:     {utc_time}")       # 2026-07-08 18:30:00+00:00
print(f"Toronto: {toronto_time}")   # 2026-07-08 14:30:00-04:00
print(f"London:  {london_time}")    # 2026-07-08 19:30:00+01:00
print(f"Mumbai:  {mumbai_time}")    # 2026-07-09 00:00:00+05:30

# Convert from local to UTC
local = datetime(2026, 7, 8, 14, 30, tzinfo=ZoneInfo("America/Toronto"))
utc = local.astimezone(ZoneInfo("UTC"))
print(f"Local: {local} → UTC: {utc}")
# Local: 2026-07-08 14:30:00-04:00 → UTC: 2026-07-08 18:30:00+00:00

UTC — The Universal Standard

Why You Should Always Store UTC

1. No ambiguity — UTC is the same everywhere. “2026-07-08 18:30:00 UTC” means one exact moment worldwide.
2. No DST surprises — UTC does not observe daylight saving time. No “spring forward” where 2:30 AM does not exist, no “fall back” where 1:30 AM happens twice.
3. Easy comparison — all timestamps in UTC can be compared directly without conversion.
4. Server-independent — your pipeline produces the same timestamps whether deployed in US East, Europe West, or India South.
5. Convert to local only for display — store UTC in your data lake, convert to the user’s timezone only in the presentation layer (Power BI, dashboards, reports).

# The standard pattern for pipeline timestamps
from datetime import datetime, timezone

# Always use UTC for storage
loaded_at = datetime.now(timezone.utc).isoformat()
print(loaded_at)   # "2026-07-08T18:30:00.123456+00:00"

# For pipeline config and watermarks — always UTC
watermark = datetime.now(timezone.utc).strftime("%Y-%m-%d %H:%M:%S")
print(watermark)   # "2026-07-08 18:30:00"

Daylight Saving Time Pitfalls

from datetime import datetime
from zoneinfo import ZoneInfo

# DST TRAP: Toronto is UTC-5 in winter, UTC-4 in summer
winter = datetime(2026, 1, 15, 12, 0, tzinfo=ZoneInfo("America/Toronto"))
summer = datetime(2026, 7, 15, 12, 0, tzinfo=ZoneInfo("America/Toronto"))

print(winter)   # 2026-01-15 12:00:00-05:00 (EST)
print(summer)   # 2026-07-15 12:00:00-04:00 (EDT)

# If you compute a timedelta across DST boundaries, zoneinfo handles it correctly
# But if you use naive datetimes, you silently get wrong results

# DST TRAP: "2:30 AM" does not exist during spring forward
# In March 2026, clocks jump from 2:00 AM to 3:00 AM
# datetime(2026, 3, 8, 2, 30, tzinfo=ZoneInfo("America/Toronto"))
# → This is handled by zoneinfo (folds/gaps)

# DST TRAP: "1:30 AM" happens TWICE during fall back
# In November 2026, clocks go from 2:00 AM back to 1:00 AM
# Which 1:30 AM do you mean? zoneinfo uses fold=0/1 to disambiguate

# SOLUTION: Store everything in UTC. Convert to local only for display.
# UTC has no DST — no gaps, no folds, no ambiguity.

ISO 8601 — The Universal Date Format

What ISO 8601 Is

ISO 8601 is the international standard for date and time representation. It is unambiguous (07/08/2026 — is that July 8 or August 7? ISO 8601: 2026-07-08 — always July 8), sortable (string sorting gives chronological order), and supported by every database, API, and programming language.

# ISO 8601 formats
# Date:          2026-07-08
# DateTime:      2026-07-08T14:30:00
# With timezone: 2026-07-08T14:30:00+00:00
# With Z (UTC):  2026-07-08T14:30:00Z

from datetime import datetime, timezone

dt = datetime(2026, 7, 8, 14, 30, 0, tzinfo=timezone.utc)

# To ISO 8601 string
print(dt.isoformat())           # "2026-07-08T14:30:00+00:00"

# From ISO 8601 string
parsed = datetime.fromisoformat("2026-07-08T14:30:00+00:00")
print(parsed)                    # 2026-07-08 14:30:00+00:00

# RECOMMENDATION: Use ISO 8601 for all date strings in your pipeline
# JSON configs, API responses, watermark values, audit logs

Working with Timestamps

Unix Timestamps (Epoch Time)

# A Unix timestamp is the number of seconds since January 1, 1970 00:00:00 UTC
# Also called "epoch time" or "POSIX time"

from datetime import datetime, timezone

# Current timestamp
ts = datetime.now(timezone.utc).timestamp()
print(ts)   # 1783706200.123456

# Convert datetime to timestamp
dt = datetime(2026, 7, 8, 18, 30, 0, tzinfo=timezone.utc)
ts = dt.timestamp()
print(ts)   # 1783706200.0

# Convert timestamp to datetime
dt = datetime.fromtimestamp(1783706200.0, tz=timezone.utc)
print(dt)   # 2026-07-08 18:30:00+00:00

# WARNING: datetime.fromtimestamp(ts) WITHOUT tz returns LOCAL time
# Always pass tz=timezone.utc

# Millisecond timestamps (common in APIs and JavaScript)
ms_timestamp = 1783706200123   # milliseconds
dt = datetime.fromtimestamp(ms_timestamp / 1000, tz=timezone.utc)
print(dt)   # 2026-07-08 18:30:00.123000+00:00

Data Engineering Patterns

Date-Partitioned File Paths

from datetime import datetime, timezone
from pathlib import Path

def get_partition_path(base_path, table_name, dt=None):
    """Generate a date-partitioned path for data lake storage."""
    if dt is None:
        dt = datetime.now(timezone.utc)

    partition = dt.strftime("year=%Y/month=%m/day=%d")
    return Path(base_path) / table_name / partition

# Usage
path = get_partition_path("/data/bronze", "customers")
print(path)   # /data/bronze/customers/year=2026/month=07/day=08

# With a specific date
path = get_partition_path("/data/bronze", "orders", datetime(2026, 1, 15))
print(path)   # /data/bronze/orders/year=2026/month=01/day=15

# ADF/Fabric expression equivalent:
# @concat('bronze/customers/year=', formatDateTime(utcnow(), 'yyyy'),
#         '/month=', formatDateTime(utcnow(), 'MM'),
#         '/day=', formatDateTime(utcnow(), 'dd'))

Watermark Pattern for Incremental Loads

from datetime import datetime, timezone, timedelta

def get_incremental_window(last_loaded, buffer_minutes=5):
    """Calculate the time window for incremental extraction.

    Args:
        last_loaded: datetime of the last successful load (UTC)
        buffer_minutes: overlap to catch late-arriving records

    Returns:
        (start, end) tuple of UTC datetimes
    """
    # Start from the last loaded time minus a buffer
    # The buffer catches records that arrived between the last load and now
    start = last_loaded - timedelta(minutes=buffer_minutes)
    end = datetime.now(timezone.utc)

    return start, end

# Usage
last_loaded = datetime(2026, 7, 7, 18, 0, 0, tzinfo=timezone.utc)
start, end = get_incremental_window(last_loaded)

print(f"Extract WHERE modified_date BETWEEN '{start}' AND '{end}'")
# Extract WHERE modified_date BETWEEN '2026-07-07 17:55:00+00:00' AND '2026-07-08 18:30:00+00:00'

# The 5-minute buffer ensures records that were being written
# during the previous load are not missed

Pipeline Scheduling Windows

from datetime import datetime, timedelta, timezone

def get_schedule_window(window_size_hours=1, reference_time=None):
    """Calculate the processing window for a scheduled pipeline.

    Returns the start and end of the current processing window.
    Example: For hourly windows, if it is 14:30, the window is 14:00-15:00.
    """
    if reference_time is None:
        reference_time = datetime.now(timezone.utc)

    # Floor to the nearest window boundary
    window_start = reference_time.replace(
        minute=0, second=0, microsecond=0
    )
    # Align to window_size boundaries
    hour = window_start.hour
    aligned_hour = (hour // window_size_hours) * window_size_hours
    window_start = window_start.replace(hour=aligned_hour)

    window_end = window_start + timedelta(hours=window_size_hours)

    return window_start, window_end

# Hourly windows
start, end = get_schedule_window(window_size_hours=1)
print(f"Window: {start} to {end}")
# Window: 2026-07-08 18:00:00+00:00 to 2026-07-08 19:00:00+00:00

# 4-hour windows
start, end = get_schedule_window(window_size_hours=4)
print(f"Window: {start} to {end}")
# Window: 2026-07-08 16:00:00+00:00 to 2026-07-08 20:00:00+00:00

Audit Timestamps

from datetime import datetime, timezone

class AuditLogger:
    """Add standardized audit timestamps to pipeline records."""

    def __init__(self, pipeline_name, run_id=None):
        self.pipeline_name = pipeline_name
        self.run_id = run_id or datetime.now(timezone.utc).strftime("%Y%m%d_%H%M%S")
        self.run_start = datetime.now(timezone.utc)

    def add_audit_columns(self, record):
        """Add audit columns to a single record."""
        record["_loaded_at"] = datetime.now(timezone.utc).isoformat()
        record["_pipeline_name"] = self.pipeline_name
        record["_run_id"] = self.run_id
        return record

    def get_run_summary(self):
        """Get timing summary for the pipeline run."""
        elapsed = datetime.now(timezone.utc) - self.run_start
        return {
            "pipeline": self.pipeline_name,
            "run_id": self.run_id,
            "started_at": self.run_start.isoformat(),
            "completed_at": datetime.now(timezone.utc).isoformat(),
            "duration_seconds": elapsed.total_seconds(),
        }

# Usage
audit = AuditLogger("daily_customer_load")
for row in data:
    audit.add_audit_columns(row)

print(audit.get_run_summary())

Date Range Generator

from datetime import date, timedelta

def date_range(start_date, end_date):
    """Generate all dates from start to end (inclusive)."""
    current = start_date
    while current <= end_date:
        yield current
        current += timedelta(days=1)

# Generate all dates in a month
for d in date_range(date(2026, 7, 1), date(2026, 7, 31)):
    print(d.strftime("%Y-%m-%d"))

# Backfill: process each day's data individually
for d in date_range(date(2026, 6, 1), date(2026, 6, 30)):
    process_day(d)

# Generate partition list for cleanup
old_dates = list(date_range(date(2026, 1, 1), date(2026, 3, 31)))
for d in old_dates:
    partition = d.strftime("year=%Y/month=%m/day=%d")
    print(f"Deleting partition: {partition}")

Business Days Calculator

from datetime import date, timedelta

def add_business_days(start_date, business_days):
    """Add N business days (skipping weekends) to a date."""
    current = start_date
    added = 0
    while added < business_days:
        current += timedelta(days=1)
        if current.weekday() < 5:   # Monday=0 through Friday=4
            added += 1
    return current

def count_business_days(start_date, end_date):
    """Count business days between two dates (exclusive of start, inclusive of end)."""
    count = 0
    current = start_date
    while current < end_date:
        current += timedelta(days=1)
        if current.weekday() < 5:
            count += 1
    return count

# SLA: pipeline must complete within 3 business days
today = date(2026, 7, 8)   # Wednesday
deadline = add_business_days(today, 3)
print(f"Deadline: {deadline}")   # 2026-07-13 (Monday — skipped Saturday and Sunday)

# How many business days until quarter end?
quarter_end = date(2026, 9, 30)
remaining = count_business_days(today, quarter_end)
print(f"Business days remaining: {remaining}")

Common Mistakes

1. Using datetime.now() without timezone — returns the server’s local time, which changes if you deploy to a different region or during DST transitions. Always use datetime.now(timezone.utc) for pipeline timestamps.
2. Confusing strftime and strptime — strftime formats (f = format, datetime → string). strptime parses (p = parse, string → datetime). Memory trick: f = format, p = parse.
3. Using timedelta.seconds instead of total_seconds() — .seconds gives only the seconds component of the last partial day, not the total. timedelta(days=2, hours=3).seconds is 10,800 (3 hours), not 183,600 (2 days + 3 hours). Always use .total_seconds().
4. Comparing naive and aware datetimes — Python raises a TypeError, which is correct behavior. But silently working with a mix of naive and aware datetimes in your pipeline means comparisons and filters give wrong results. Standardize: all UTC, all aware.
5. Hardcoding UTC offset instead of using timezone names — timedelta(hours=-5) is EST but does not account for DST. ZoneInfo("America/Toronto") automatically handles EST/EDT transitions. Always use timezone names, not offsets.
6. Not handling date parsing errors — strptime on a bad date string throws ValueError. In a pipeline processing millions of rows, one bad date crashes everything. Always wrap parsing in try/except with a fallback.
7. Using datetime.utcnow() — this function returns a naive datetime (no timezone info) even though it is UTC. Python 3.12+ deprecates it. Use datetime.now(timezone.utc) instead, which returns an aware datetime.

Interview Questions

Q: What is the difference between a naive and an aware datetime? A: A naive datetime has no timezone information (tzinfo=None). It represents a local time but you do not know which timezone. An aware datetime includes timezone info and represents an unambiguous moment in time. For data pipelines, always use aware datetimes in UTC to avoid timezone-related bugs when comparing timestamps across regions.

Q: What is the difference between strftime and strptime? A: strftime (f = format) converts a datetime object into a formatted string. strptime (p = parse) converts a string into a datetime object. They use the same format codes (%Y, %m, %d, etc.) but in opposite directions. Example: dt.strftime("%Y-%m-%d") produces “2026-07-08”. datetime.strptime("2026-07-08", "%Y-%m-%d") produces a datetime object.

Q: Why should you store timestamps in UTC? A: UTC is universal (same everywhere), has no daylight saving time transitions (no gaps or duplicates), enables direct comparison without conversion, and makes your pipeline server-independent. Store in UTC, convert to local timezone only in the presentation layer (dashboards, reports). This prevents bugs when the pipeline runs in a different region or across DST boundaries.

Q: What is a timedelta and what is the difference between .seconds and .total_seconds()? A: A timedelta represents a duration (e.g., 3 days, 5 hours). .seconds returns only the seconds component of the last partial day — for timedelta(days=2, hours=3), it returns 10,800 (3 hours in seconds), ignoring the 2 days. .total_seconds() returns the complete duration in seconds — 183,600 (2 days + 3 hours). Always use .total_seconds() for duration calculations.

Q: How do you handle multiple date formats from different source systems? A: Create a multi-format parser that tries each expected format in order using strptime inside a try/except loop. Return the first successful parse. If no format matches, return None or raise an error. This handles sources that send dates as “2026-07-08”, “07/08/2026”, “08-Jul-2026”, or “July 8, 2026” — all in the same pipeline.

Q: What is ISO 8601 and why is it important for data engineering? A: ISO 8601 is the international standard for date/time representation: 2026-07-08T14:30:00+00:00. It is unambiguous (no month/day confusion), sortable as a string (alphabetical = chronological), and supported universally by databases, APIs, JSON, and every programming language. Use it for all date strings in configs, API responses, watermarks, and audit logs.

Wrapping Up

Dates and times are deceptively simple until timezones, DST, and multiple source formats enter the picture. The datetime module gives you four tools: date for calendar dates, time for time-of-day, datetime for combined date-and-time, and timedelta for durations and date math. strftime formats datetimes into strings (f = format). strptime parses strings into datetimes (p = parse). zoneinfo handles timezones correctly, including DST.

For data engineering, the rules are clear: store everything in UTC, use ISO 8601 for all date strings, use aware datetimes with timezone.utc, parse source dates with error handling and multi-format support, and use total_seconds() (never .seconds) for duration calculations. These patterns — date-partitioned paths, watermark windows, audit timestamps, and date range generators — will appear in every production pipeline you build.

Previous in this series: Modules, Packages, Virtual Environments, and Imports

Next in this series: Regular Expressions — Pattern Matching and Data Cleaning

---

Naveen Vuppula is a Senior Data Engineering Consultant and app developer based in Ontario, Canada. He writes about Python, SQL, AWS, Azure, and everything data engineering at DriveDataScience.com.