Python OOP Advanced: Inheritance, Method Overriding, super(), Multiple Inheritance, MRO, Polymorphism, Abstract Classes, Mixins, Composition vs Inheritance, and Every Pattern Data Engineers Need

In the previous post, you learned to build classes — blueprints for objects that bundle data and behavior. But what happens when you need 10 different types of pipelines that all share 80% of the same logic? Do you copy-paste the code into 10 classes? That is a maintenance nightmare — fix a bug in one, forget the other nine.

Inheritance solves this. You write the shared logic once in a parent class, and each specialized class inherits it and adds only what is different. Polymorphism lets you treat all 10 pipeline types interchangeably — call .run() on any of them without knowing or caring which specific type it is. Abstract classes enforce a contract — “every pipeline MUST implement an extract() method, or Python refuses to create it.”

Real-life analogy: Think of a vehicle registration system. Every vehicle — car, truck, motorcycle, bus — has a registration number, an owner, and a year of manufacture. That is the parent class (Vehicle). But each vehicle type has its own specifics: a truck has a payload capacity, a motorcycle has an engine displacement category, a bus has a passenger capacity. Each type inherits the shared fields and adds its own. When the DMV processes registrations, it calls vehicle.calculate_annual_fee() — and each type calculates its fee differently (polymorphism). The DMV does not care whether it is processing a car or a bus — the method name is the same, but the behavior adapts. And the DMV requires that every vehicle type MUST implement calculate_annual_fee() before it can be registered — that is an abstract class.

Table of Contents

• Inheritance Basics
• What Inheritance Is
• Your First Parent-Child Class
• What Gets Inherited
• Method Overriding
• Overriding a Parent Method
• When to Override
• super() — Calling the Parent
• What super() Does
• super().__init__() — Extending the Constructor
• super() in Other Methods
• isinstance() and issubclass()
• Type Checking with Inheritance
• Multiple Inheritance
• Inheriting from Multiple Parents
• The Diamond Problem
• Method Resolution Order (MRO)
• Polymorphism
• What Polymorphism Means
• Polymorphism in Action
• Duck Typing
• Polymorphism with Built-in Functions
• Abstract Classes (abc Module)
• What Abstract Classes Are
• Creating Abstract Classes
• Abstract Properties
• Why Abstract Classes Matter
• Mixins
• What Mixins Are
• Logging Mixin
• Serialization Mixin
• Combining Multiple Mixins
• Composition vs Inheritance
• When Inheritance Goes Wrong
• Composition: “Has-a” Instead of “Is-a”
• When to Use Each
• Data Engineering Patterns
• Abstract Pipeline Base Class
• Source-Specific Pipeline Implementations
• Polymorphic Pipeline Runner
• Validator Hierarchy
• Composed Pipeline with Pluggable Components
• Common Mistakes
• Interview Questions
• Wrapping Up

Inheritance Basics

What Inheritance Is

Inheritance lets a new class (child/subclass) reuse all the attributes and methods of an existing class (parent/superclass) without rewriting them. The child class can then add new features or override existing ones to change behavior.

Real-life analogy: Think of a smartphone. The original iPhone defined the blueprint — touchscreen, apps, camera, phone calls. Every new iPhone model inherits these core features. But the iPhone 16 adds new features (action button, USB-C) and overrides old ones (better camera, faster chip). Apple did not rebuild the phone from scratch — they inherited the blueprint and improved specific parts. That is inheritance.

Your First Parent-Child Class

# PARENT class (also called base class or superclass)
class DataSource:
    """Base class for all data sources."""

    def __init__(self, name, connection_string):
        self.name = name
        self.connection_string = connection_string
        self.is_connected = False
        self.row_count = 0

    def connect(self):
        """Connect to the data source."""
        self.is_connected = True
        print(f"Connected to {self.name}")

    def disconnect(self):
        """Disconnect from the data source."""
        self.is_connected = False
        print(f"Disconnected from {self.name}")

    def summary(self):
        status = "connected" if self.is_connected else "disconnected"
        return f"{self.name} ({status}): {self.row_count:,} rows"


# CHILD class (also called derived class or subclass)
# Inherits EVERYTHING from DataSource and adds SQL-specific features
class SQLSource(DataSource):
    """A SQL database data source — inherits from DataSource."""

    def __init__(self, name, connection_string, database, schema="dbo"):
        # Call the PARENT's __init__ to set up name, connection_string, etc.
        super().__init__(name, connection_string)
        # Add child-specific attributes
        self.database = database
        self.schema = schema

    def execute_query(self, sql):
        """Child-specific method — only SQL sources can execute queries."""
        if not self.is_connected:
            raise ConnectionError(f"Not connected to {self.name}")
        print(f"Executing on {self.database}.{self.schema}: {sql}")
        # ... execute and return results


# Usage
sql = SQLSource("Azure SQL", "server.database.windows.net", "warehouse", "bronze")

# Inherited methods — defined in DataSource, available in SQLSource
sql.connect()           # "Connected to Azure SQL"
print(sql.summary())    # "Azure SQL (connected): 0 rows"

# Child-specific methods
sql.execute_query("SELECT * FROM customers")

# Inherited attributes
print(sql.name)              # "Azure SQL" — from DataSource
print(sql.is_connected)      # True — from DataSource

# Child-specific attributes
print(sql.database)          # "warehouse" — from SQLSource
print(sql.schema)            # "bronze" — from SQLSource

What Gets Inherited

Inherited?	What	Example
✅ Yes	All instance methods	connect(), disconnect(), summary()
✅ Yes	All class methods and static methods	@classmethod, @staticmethod
✅ Yes	All class attributes	version = "2.0"
✅ Yes	Properties (@property)	Getter, setter, deleter
✅ Yes	Dunder methods	__str__, __repr__, __len__
⚠️ Partially	Instance attributes	Only if child calls super().__init__()
❌ No	Nothing is excluded by default	Python inherits everything — override to change

Method Overriding

Overriding a Parent Method

When a child class defines a method with the same name as a parent method, the child’s version takes over. The parent’s version is hidden (but not deleted — you can still access it via super()). This is how you customize inherited behavior without modifying the parent class.

class DataSource:
    def __init__(self, name):
        self.name = name
        self.row_count = 0

    def connect(self):
        print(f"Generic connection to {self.name}")

    def summary(self):
        return f"Source: {self.name}, Rows: {self.row_count:,}"


class SQLSource(DataSource):
    def __init__(self, name, database):
        super().__init__(name)
        self.database = database

    # OVERRIDE connect — SQL needs a specific connection flow
    def connect(self):
        print(f"Establishing JDBC connection to {self.database} on {self.name}")
        print(f"Authenticating with service principal...")
        print(f"Connected to {self.database}")

    # OVERRIDE summary — include database-specific info
    def summary(self):
        return f"SQL Source: {self.name}/{self.database}, Rows: {self.row_count:,}"


class APISource(DataSource):
    def __init__(self, name, base_url):
        super().__init__(name)
        self.base_url = base_url

    def connect(self):
        print(f"Testing API endpoint: {self.base_url}/health")
        print(f"API {self.name} is reachable")

    def summary(self):
        return f"API Source: {self.name} ({self.base_url}), Rows: {self.row_count:,}"


# Each source type connects differently — same method name, different behavior
sql = SQLSource("Azure SQL", "warehouse")
api = APISource("Customer API", "https://api.example.com")

sql.connect()
# Establishing JDBC connection to warehouse on Azure SQL
# Authenticating with service principal...
# Connected to warehouse

api.connect()
# Testing API endpoint: https://api.example.com/health
# API Customer API is reachable

Real-life analogy: Every airline checks you in for a flight (check_in()), but each airline does it differently. Delta uses an app, Emirates has a lounge counter, and a budget airline might use a kiosk. The action is the same — “check in” — but the implementation varies. That is method overriding. The parent class (Airline) defines check_in(), and each child (Delta, Emirates, BudgetAir) overrides it with their own process.

When to Override

• Override completely — when the child needs entirely different logic. The API source connects via HTTP; the SQL source connects via JDBC. No shared code.
• Override and extend — when the child needs the parent’s logic PLUS additional steps. Use super() to call the parent’s version first, then add your own logic.
• Do not override — when the parent’s logic is fine for the child. disconnect() might be the same for all sources.

super() — Calling the Parent

What super() Does

super() gives you a reference to the parent class, letting you call its methods from inside the child. This is essential when you want to extend parent behavior rather than replace it entirely. Think of it as “do everything my parent does, AND ALSO do this extra thing.”

super().__init__() — Extending the Constructor

# The most common use of super() — calling the parent's __init__

class DataSource:
    def __init__(self, name, connection_string):
        self.name = name
        self.connection_string = connection_string
        self.is_connected = False
        self.row_count = 0
        self.errors = []

class SQLSource(DataSource):
    def __init__(self, name, connection_string, database, schema="dbo"):
        # FIRST: call parent's __init__ to set up shared attributes
        super().__init__(name, connection_string)
        # THEN: set up child-specific attributes
        self.database = database
        self.schema = schema
        self.query_count = 0

# What happens without super().__init__():
class BrokenSource(DataSource):
    def __init__(self, name, database):
        # Forgot super().__init__()!
        self.database = database

broken = BrokenSource("test", "mydb")
# broken.name         → AttributeError! name was never set
# broken.is_connected → AttributeError! is_connected was never set
# The parent's __init__ was NEVER called — no shared attributes exist

# RULE: Always call super().__init__() in the child's __init__
# unless you deliberately want to skip parent initialization (very rare)

Real-life analogy: super().__init__() is like filling out a new-hire form. The company has a standard form (parent) that every employee fills out — name, address, tax ID. Each department (child) adds its own section — Engineering adds GitHub username, Sales adds territory. But you MUST fill out the standard section first. Skipping super().__init__() is like submitting only the department section — HR rejects it because the basics are missing.

super() in Other Methods

# Use super() when you want PARENT behavior + EXTRA behavior

class DataSource:
    def connect(self):
        self.is_connected = True
        print(f"Base connection established to {self.name}")

    def validate(self):
        """Basic validation that applies to all sources."""
        if not self.is_connected:
            raise ConnectionError("Must connect before validating")
        print(f"Basic validation passed for {self.name}")

class SQLSource(DataSource):
    def connect(self):
        # Do the parent's connection FIRST
        super().connect()
        # THEN add SQL-specific setup
        self.cursor = self._create_cursor()
        print(f"SQL cursor created for {self.database}")

    def validate(self):
        # Do ALL parent validation first
        super().validate()
        # THEN add SQL-specific validation
        if not self.schema:
            raise ValueError("Schema must be specified for SQL sources")
        print(f"SQL schema validation passed: {self.schema}")

sql = SQLSource("Azure SQL", "conn_str", "warehouse")
sql.connect()
# Base connection established to Azure SQL     ← from parent (super)
# SQL cursor created for warehouse              ← from child

sql.validate()
# Basic validation passed for Azure SQL         ← from parent (super)
# SQL schema validation passed: dbo             ← from child

isinstance() and issubclass()

Type Checking with Inheritance

class DataSource: pass
class SQLSource(DataSource): pass
class APISource(DataSource): pass

sql = SQLSource()
api = APISource()

# isinstance() checks if an object is an instance of a class (or its parents)
isinstance(sql, SQLSource)    # True — sql IS a SQLSource
isinstance(sql, DataSource)   # True — sql IS ALSO a DataSource (via inheritance)
isinstance(sql, APISource)    # False — sql is NOT an APISource

# issubclass() checks class relationships (not instances)
issubclass(SQLSource, DataSource)   # True — SQLSource inherits from DataSource
issubclass(APISource, DataSource)   # True
issubclass(SQLSource, APISource)    # False — no relationship between siblings

# Practical use: type checking in functions
def process_source(source):
    if not isinstance(source, DataSource):
        raise TypeError(f"Expected DataSource, got {type(source).__name__}")
    source.connect()
    return source.extract()

# Works with any DataSource subclass — SQLSource, APISource, etc.
process_source(sql)   # ✅ Works
process_source(api)   # ✅ Works
process_source("hi")  # ❌ TypeError

Multiple Inheritance

Inheriting from Multiple Parents

Python allows a class to inherit from multiple parents. The child gets all attributes and methods from all parents. This is powerful but can be confusing — use it carefully.

# Two independent parent classes
class Connectable:
    """Mixin: provides connect/disconnect behavior."""
    def __init__(self):
        self.is_connected = False

    def connect(self):
        self.is_connected = True
        print("Connected")

    def disconnect(self):
        self.is_connected = False
        print("Disconnected")

class Loggable:
    """Mixin: provides logging behavior."""
    def __init__(self):
        self._log = []

    def log(self, message):
        self._log.append(message)
        print(f"LOG: {message}")

    def get_logs(self):
        return self._log

# Child inherits from BOTH parents
class DatabasePipeline(Connectable, Loggable):
    def __init__(self, name):
        Connectable.__init__(self)    # Initialize Connectable
        Loggable.__init__(self)       # Initialize Loggable
        self.name = name

    def run(self):
        self.log(f"Pipeline {self.name} starting")
        self.connect()
        self.log(f"Connected — running pipeline")
        # ... do work
        self.disconnect()
        self.log(f"Pipeline {self.name} complete")

pipe = DatabasePipeline("daily_load")
pipe.run()
# LOG: Pipeline daily_load starting
# Connected
# LOG: Connected — running pipeline
# Disconnected
# LOG: Pipeline daily_load complete

print(pipe.get_logs())
# ['Pipeline daily_load starting', 'Connected — running pipeline', 'Pipeline daily_load complete']

The Diamond Problem

The diamond problem occurs when a class inherits from two parents that share a common grandparent. Which version of the grandparent’s method should the child use? Python solves this with the Method Resolution Order (MRO).

#         DataSource          ← grandparent
#          /      #    SQLSource   FileSource   ← parents (both inherit from DataSource)
#          \      /
#       HybridSource          ← child (diamond — which DataSource.__init__?)

class DataSource:
    def __init__(self, name):
        print(f"DataSource.__init__({name})")
        self.name = name

class SQLSource(DataSource):
    def __init__(self, name, database):
        print(f"SQLSource.__init__({name}, {database})")
        super().__init__(name)       # Calls next in MRO, NOT necessarily DataSource
        self.database = database

class FileSource(DataSource):
    def __init__(self, name, filepath):
        print(f"FileSource.__init__({name}, {filepath})")
        super().__init__(name)       # Calls next in MRO
        self.filepath = filepath

class HybridSource(SQLSource, FileSource):
    def __init__(self, name, database, filepath):
        print(f"HybridSource.__init__({name})")
        super().__init__(name, database)   # Follows MRO
        self.filepath = filepath

# Without super() and MRO, DataSource.__init__ would be called TWICE
# Python's MRO ensures each class's __init__ is called exactly ONCE

Method Resolution Order (MRO)

# MRO determines the order Python searches for methods
# It uses the C3 linearization algorithm

# Check a class's MRO:
print(HybridSource.__mro__)
# (HybridSource, SQLSource, FileSource, DataSource, object)

# Or more readable:
print(HybridSource.mro())
# [HybridSource, SQLSource, FileSource, DataSource, object]

# When you call hybrid.connect():
# Python checks: HybridSource → SQLSource → FileSource → DataSource → object
# It uses the FIRST match it finds

# The MRO follows three rules:
# 1. A child comes before its parents
# 2. Parents are checked left to right (order in the class definition)
# 3. Each class appears exactly once

# This is why super() works correctly in the diamond pattern:
# super() follows the MRO, not just the direct parent

Real-life analogy: MRO is like a chain of command in a corporation. When an employee has a question, they ask their immediate manager first. If the manager does not know, it goes up to the director, then the VP, then the CEO. In a matrix organization (multiple inheritance), the chain follows a specific order that ensures every level is consulted exactly once — no one is asked twice, and no one is skipped.

Polymorphism

What Polymorphism Means

Polymorphism means “many forms.” It lets you call the same method on different types and each type responds in its own way. The caller does not need to know the specific type — it just calls the method and trusts the object to do the right thing.

Real-life analogy: Think of a universal remote control. You press the “Power” button, and every device responds differently — the TV turns on, the soundbar starts up, the Blu-ray player begins loading. You do not need a separate remote for each device. The button is the same (.power_on()), but each device handles it differently. That is polymorphism — one interface, many implementations.

Polymorphism in Action

# Three different source types — each extracts data differently
class SQLSource:
    def __init__(self, table_name, connection_string):
        self.table_name = table_name
        self.connection_string = connection_string

    def extract(self):
        print(f"Running SQL: SELECT * FROM {self.table_name}")
        return [{"id": 1, "name": "Naveen"}, {"id": 2, "name": "Alice"}]

    def source_info(self):
        return f"SQL: {self.table_name}"

class APISource:
    def __init__(self, url, endpoint):
        self.url = url
        self.endpoint = endpoint

    def extract(self):
        print(f"GET {self.url}/{self.endpoint}")
        return [{"id": 1, "name": "Bob"}]

    def source_info(self):
        return f"API: {self.url}/{self.endpoint}"

class FileSource:
    def __init__(self, filepath):
        self.filepath = filepath

    def extract(self):
        print(f"Reading file: {self.filepath}")
        return [{"id": 1, "name": "Charlie"}]

    def source_info(self):
        return f"File: {self.filepath}"


# POLYMORPHISM: This function works with ANY source type
# It does not know or care whether it is SQL, API, or File
def load_from_source(source):
    """Works with any object that has extract() and source_info()."""
    print(f"Loading from: {source.source_info()}")
    data = source.extract()
    print(f"Loaded {len(data)} records")
    return data

# All three types work with the SAME function — polymorphism
sources = [
    SQLSource("customers", "server.db.windows.net"),
    APISource("https://api.example.com", "users"),
    FileSource("/data/products.csv"),
]

for source in sources:
    load_from_source(source)
    print()

# Loading from: SQL: customers
# Running SQL: SELECT * FROM customers
# Loaded 2 records
#
# Loading from: API: https://api.example.com/users
# GET https://api.example.com/users
# Loaded 1 records
#
# Loading from: File: /data/products.csv
# Reading file: /data/products.csv
# Loaded 1 records

Duck Typing

Python does not require inheritance for polymorphism. If an object has the right methods, it works — regardless of its class hierarchy. This is called duck typing: “If it walks like a duck and quacks like a duck, it is a duck.”

# These three classes have NO common parent — but they all have extract()
class SQLSource:
    def extract(self): return [1, 2, 3]

class APISource:
    def extract(self): return [4, 5, 6]

class MockSource:   # Not even a real source — a test fake
    def extract(self): return [7, 8, 9]

# This function works with ALL three — no shared parent needed
def process(source):
    data = source.extract()   # Python does not check the type — it checks the METHOD
    return len(data)

# All three work because they all have extract()
print(process(SQLSource()))    # 3
print(process(APISource()))    # 3
print(process(MockSource()))   # 3

# Duck typing in the standard library:
# - for loops work on anything with __iter__
# - len() works on anything with __len__
# - print() works on anything with __str__
# - with statement works on anything with __enter__ and __exit__

Real-life analogy: You call a taxi. You do not ask the driver “Are you a certified subclass of the TransportProvider class?” You just get in, say where you are going, and expect them to drive you there. If they can drive you there, they are a taxi — regardless of whether they are Uber, Lyft, or a traditional cab. Python works the same way — it does not check the family tree, it checks the capability.

Polymorphism with Built-in Functions

# Python's built-in functions are polymorphic — they work with many types

# len() works on strings, lists, dicts, AND your custom objects
print(len("hello"))       # 5 — str.__len__()
print(len([1, 2, 3]))     # 3 — list.__len__()
print(len({"a": 1}))      # 1 — dict.__len__()

class DataBatch:
    def __init__(self, records):
        self.records = records
    def __len__(self):
        return len(self.records)

batch = DataBatch([1, 2, 3, 4, 5])
print(len(batch))          # 5 — DataBatch.__len__()

# + operator works on ints, floats, strings, lists — all via __add__
print(1 + 2)               # 3 — int.__add__()
print("hello " + "world")  # "hello world" — str.__add__()
print([1, 2] + [3, 4])     # [1, 2, 3, 4] — list.__add__()

# Same function, different behavior based on the TYPE — polymorphism

Abstract Classes (abc Module)

What Abstract Classes Are

An abstract class is a class that cannot be instantiated directly. It defines a contract — a set of methods that every subclass MUST implement. If a subclass forgets to implement an abstract method, Python raises a TypeError at instantiation time, not at runtime. This catches missing implementations early.

Real-life analogy: An abstract class is like a building code. The code says “every building MUST have an emergency exit, a fire suppression system, and structural load calculations.” The code itself is not a building — you cannot live in it. But every building (concrete class) MUST satisfy the code requirements before it can be approved. If the architect forgets the emergency exit, the inspection fails before anyone moves in (TypeError at instantiation, not a runtime crash when the fire alarm goes off).

Creating Abstract Classes

from abc import ABC, abstractmethod

class DataSource(ABC):
    """Abstract base class — cannot be instantiated directly.
    Every subclass MUST implement extract() and get_schema()."""

    def __init__(self, name):
        self.name = name
        self.row_count = 0

    @abstractmethod
    def extract(self):
        """Subclasses MUST implement this — defines HOW to get data."""
        pass

    @abstractmethod
    def get_schema(self):
        """Subclasses MUST return the schema of the data source."""
        pass

    # Non-abstract methods are inherited normally
    def summary(self):
        return f"{self.name}: {self.row_count:,} rows"

# ❌ Cannot instantiate the abstract class directly
try:
    source = DataSource("test")
except TypeError as e:
    print(e)
    # Can't instantiate abstract class DataSource with abstract methods
    # 'extract', 'get_schema'


# ✅ Concrete class — implements ALL abstract methods
class SQLSource(DataSource):
    def __init__(self, name, table):
        super().__init__(name)
        self.table = table

    def extract(self):
        """REQUIRED: implements the abstract method."""
        print(f"SELECT * FROM {self.table}")
        self.row_count = 50000
        return [{"id": 1}]

    def get_schema(self):
        """REQUIRED: implements the abstract method."""
        return {"id": "int", "name": "varchar(100)", "email": "varchar(200)"}

sql = SQLSource("Azure SQL", "customers")   # ✅ Works — all abstract methods implemented
data = sql.extract()
print(sql.summary())   # "Azure SQL: 50,000 rows" — inherited from DataSource


# ❌ Incomplete class — forgets one abstract method
class BrokenSource(DataSource):
    def extract(self):
        return []
    # Missing get_schema()!

try:
    broken = BrokenSource("test")
except TypeError as e:
    print(e)
    # Can't instantiate abstract class BrokenSource with abstract method 'get_schema'
    # Python catches the missing method at CREATION time, not at call time

Abstract Properties

from abc import ABC, abstractmethod

class Pipeline(ABC):
    """Every pipeline must define a source_type and target_type."""

    @property
    @abstractmethod
    def source_type(self):
        """Must return the source type string."""
        pass

    @property
    @abstractmethod
    def target_type(self):
        """Must return the target type string."""
        pass

    @abstractmethod
    def run(self):
        pass

class SQLToLakehousePipeline(Pipeline):
    @property
    def source_type(self):
        return "Azure SQL"

    @property
    def target_type(self):
        return "Fabric Lakehouse"

    def run(self):
        print(f"Loading from {self.source_type} to {self.target_type}")

pipe = SQLToLakehousePipeline()
print(pipe.source_type)   # "Azure SQL"
print(pipe.target_type)   # "Fabric Lakehouse"
pipe.run()                # "Loading from Azure SQL to Fabric Lakehouse"

Why Abstract Classes Matter

1. Enforce contracts — every subclass MUST implement certain methods. No “I forgot to add extract()” bugs at runtime.
2. Catch errors early — TypeError at instantiation, not a confusing AttributeError mid-pipeline at 3 AM.
3. Document the interface — the abstract methods tell developers exactly what they need to implement for a new source type.
4. Enable polymorphism safely — you can call source.extract() on any DataSource subclass, knowing the method exists.
5. Shared code stays in the parent — non-abstract methods like summary() are written once and inherited by all.

Mixins

What Mixins Are

A mixin is a small, focused class that provides a single capability to be “mixed in” to other classes via multiple inheritance. Mixins are not meant to be instantiated on their own — they add a feature (logging, serialization, validation) that many unrelated classes might need.

Real-life analogy: A mixin is like a phone case accessory. A wallet case adds card storage to any phone. A battery case adds extra power. A kickstand case adds a stand. The phone is the main class; the case is the mixin that adds one specific feature. You can combine multiple cases (mixins) — a wallet+kickstand case adds both features. But you would never use the case without a phone.

Logging Mixin

import logging

class LoggingMixin:
    """Adds a logger to any class. Access via self.logger."""

    @property
    def logger(self):
        # Creates a logger named after the actual class (not LoggingMixin)
        if not hasattr(self, "_logger"):
            self._logger = logging.getLogger(
                f"{self.__class__.__module__}.{self.__class__.__name__}"
            )
        return self._logger

# Any class can "mix in" logging
class CustomerPipeline(LoggingMixin):
    def __init__(self, table_name):
        self.table_name = table_name

    def run(self):
        self.logger.info(f"Starting pipeline for {self.table_name}")
        # ... pipeline logic
        self.logger.info(f"Pipeline complete for {self.table_name}")

class DataValidator(LoggingMixin):
    def validate(self, df):
        self.logger.info(f"Validating {len(df)} rows")
        # ... validation logic

# Both classes get logging without duplicating the setup
pipe = CustomerPipeline("customers")
pipe.run()
# INFO | pipeline.CustomerPipeline | Starting pipeline for customers

Serialization Mixin

import json
from datetime import datetime

class JSONSerializableMixin:
    """Adds to_json() and from_json() to any class."""

    def to_dict(self):
        """Convert object attributes to a dictionary."""
        result = {}
        for key, value in self.__dict__.items():
            if key.startswith("_"):
                continue   # Skip private attributes
            if isinstance(value, datetime):
                result[key] = value.isoformat()
            else:
                result[key] = value
        return result

    def to_json(self, indent=2):
        """Serialize the object to a JSON string."""
        return json.dumps(self.to_dict(), indent=indent)

    @classmethod
    def from_dict(cls, data):
        """Create an object from a dictionary."""
        obj = cls.__new__(cls)
        obj.__dict__.update(data)
        return obj

class PipelineResult(JSONSerializableMixin):
    def __init__(self, table_name, row_count, status, duration):
        self.table_name = table_name
        self.row_count = row_count
        self.status = status
        self.duration = duration
        self.timestamp = datetime.now()

result = PipelineResult("customers", 50000, "success", 3.2)
print(result.to_json())
# {
#   "table_name": "customers",
#   "row_count": 50000,
#   "status": "success",
#   "duration": 3.2,
#   "timestamp": "2026-07-08T14:30:00.123456"
# }

Combining Multiple Mixins

# A class can mix in MULTIPLE capabilities

class TimestampMixin:
    """Adds created_at and updated_at tracking."""
    def __init_timestamp__(self):
        self.created_at = datetime.now()
        self.updated_at = datetime.now()

    def touch(self):
        self.updated_at = datetime.now()

class CustomerPipeline(LoggingMixin, JSONSerializableMixin, TimestampMixin):
    """Has logging + JSON serialization + timestamp tracking."""

    def __init__(self, table_name):
        self.table_name = table_name
        self.__init_timestamp__()

    def run(self):
        self.logger.info(f"Running {self.table_name}")
        self.touch()    # Update the timestamp
        # ... pipeline logic
        return self.to_json()   # Return result as JSON

# One class, three mixed-in capabilities — all reusable
pipe = CustomerPipeline("customers")
pipe.run()

Composition vs Inheritance

When Inheritance Goes Wrong

# Deep inheritance hierarchies become fragile and hard to understand

# ❌ Over-inheritance — too many levels
class DataSource: pass
class CloudDataSource(DataSource): pass
class AzureDataSource(CloudDataSource): pass
class AzureSQLDataSource(AzureDataSource): pass
class AzureSQLServerlessDataSource(AzureSQLDataSource): pass
# 5 levels deep — changing DataSource can break everything below
# Each level adds tiny differences — most code is in the parents
# Debugging requires reading 5 class files

# ❌ Wrong relationship — "is-a" does not fit
class Database:
    def connect(self): pass
    def query(self): pass

class Report(Database):   # ❌ A report is NOT a database!
    def generate(self): pass
    # Report inherits connect() and query() — makes no sense

# A Report USES a database, it does not inherit from it
# This calls for COMPOSITION, not inheritance

Composition: “Has-a” Instead of “Is-a”

Inheritance models an “is-a” relationship: a SQLSource is a DataSource. Composition models a “has-a” relationship: a Pipeline has a connection, has a validator, has a logger. When in doubt, prefer composition — it is more flexible and easier to change.

# ✅ COMPOSITION — the pipeline HAS components (pluggable)

class Extractor:
    """Responsible for extracting data."""
    def __init__(self, connection_string):
        self.connection_string = connection_string

    def extract(self, table_name):
        print(f"Extracting {table_name}")
        return [{"id": 1, "name": "Naveen"}]

class Transformer:
    """Responsible for transforming data."""
    def transform(self, data, rules):
        print(f"Transforming {len(data)} records")
        return data

class Loader:
    """Responsible for loading data."""
    def __init__(self, target_connection):
        self.target_connection = target_connection

    def load(self, data, target_table):
        print(f"Loading {len(data)} records to {target_table}")

class Validator:
    """Responsible for validating data."""
    def validate(self, data, checks):
        print(f"Validating {len(data)} records against {len(checks)} checks")
        return True


# Pipeline COMPOSES these components — it does not inherit from them
class ETLPipeline:
    def __init__(self, extractor, transformer, loader, validator=None):
        self.extractor = extractor
        self.transformer = transformer
        self.loader = loader
        self.validator = validator

    def run(self, table_name, target_table, transform_rules=None, checks=None):
        # Extract
        data = self.extractor.extract(table_name)

        # Transform
        data = self.transformer.transform(data, transform_rules or [])

        # Validate (optional)
        if self.validator and checks:
            self.validator.validate(data, checks)

        # Load
        self.loader.load(data, target_table)
        print(f"Pipeline complete: {table_name} → {target_table}")


# Assemble the pipeline from components — plug and play
pipeline = ETLPipeline(
    extractor=Extractor("server.database.windows.net"),
    transformer=Transformer(),
    loader=Loader("lakehouse-connection"),
    validator=Validator()
)

pipeline.run("customers", "silver.customers")

# Swap components easily — no inheritance changes needed
# Want S3 instead of Azure SQL? Just change the Extractor
# Want to skip validation? Pass validator=None
# Want a different transform logic? Swap the Transformer

Real-life analogy: Inheritance is like a family tree — you are born with your parents’ traits and cannot change them. Composition is like building a PC — you choose the CPU, GPU, RAM, and storage independently. Want more RAM? Swap the module. Want a better GPU? Replace it. Each component is independent and replaceable. Composition gives you the same flexibility in code.

When to Use Each

Use	Inheritance (“is-a”)	Composition (“has-a”)
When	True type relationship	Pluggable behavior
Example	SQLSource is a DataSource	Pipeline has an Extractor
Levels	Keep to 2-3 levels max	No depth limit
Flexibility	Fixed at class definition	Swappable at runtime
Coupling	Tight — changes in parent affect children	Loose — components are independent
Testing	Harder to test in isolation	Easy to mock/stub each component
Best for	Shared interface + shared behavior	Shared interface + different behavior

Rule of thumb: If you are asking “Should X inherit from Y?”, ask instead “Is X truly a type of Y, or does X just use Y?” If X uses Y, compose. If X truly is Y, inherit. When in doubt, compose — it is almost always the safer choice.

Data Engineering Patterns

Abstract Pipeline Base Class

from abc import ABC, abstractmethod
import logging
import time

class BasePipeline(ABC):
    """Abstract base class for all pipelines.
    Subclasses MUST implement extract() and transform().
    load() has a default implementation that can be overridden."""

    def __init__(self, name, config):
        self.name = name
        self.config = config
        self.logger = logging.getLogger(f"pipeline.{name}")
        self.row_count = 0
        self.duration = 0.0
        self.status = "pending"

    @abstractmethod
    def extract(self):
        """REQUIRED: each pipeline defines its own extraction logic."""
        pass

    @abstractmethod
    def transform(self, data):
        """REQUIRED: each pipeline defines its own transformation logic."""
        pass

    def load(self, data, target):
        """DEFAULT: can be overridden if needed."""
        self.logger.info(f"Loading {len(data):,} rows to {target}")
        write_to_target(data, target)

    def validate(self, data):
        """DEFAULT: basic validation. Override for custom checks."""
        if not data:
            raise ValueError(f"No data extracted for {self.name}")
        return True

    def run(self):
        """Template method — defines the pipeline flow.
        Subclasses customize extract() and transform()."""
        start = time.time()
        self.logger.info(f"Pipeline '{self.name}' STARTED")

        try:
            # Step 1: Extract (abstract — each pipeline implements differently)
            data = self.extract()
            self.row_count = len(data)
            self.logger.info(f"Extracted {self.row_count:,} rows")

            # Step 2: Validate
            self.validate(data)

            # Step 3: Transform (abstract)
            data = self.transform(data)
            self.logger.info(f"Transformed to {len(data):,} rows")

            # Step 4: Load
            target = f"{self.config.get('target_schema', 'silver')}.{self.name}"
            self.load(data, target)

            self.status = "success"
        except Exception as e:
            self.status = "failed"
            self.logger.error(f"Pipeline failed: {e}")
            raise
        finally:
            self.duration = time.time() - start
            self.logger.info(
                f"Pipeline '{self.name}' {self.status.upper()} "
                f"({self.row_count:,} rows, {self.duration:.1f}s)"
            )

Source-Specific Pipeline Implementations

# Each source type implements extract() and transform() differently

class SQLPipeline(BasePipeline):
    """Pipeline that extracts from a SQL database."""

    def extract(self):
        table = self.config["source_table"]
        conn = self.config["connection_string"]
        self.logger.info(f"Extracting from SQL: {table}")
        return query_database(conn, f"SELECT * FROM {table}")

    def transform(self, data):
        self.logger.info("Applying SQL-specific transformations")
        # Add audit columns, cast types, handle nulls
        for row in data:
            row["_loaded_at"] = datetime.now().isoformat()
            row["_source"] = "sql"
        return data


class APIPipeline(BasePipeline):
    """Pipeline that extracts from a REST API."""

    def extract(self):
        url = self.config["api_url"]
        self.logger.info(f"Extracting from API: {url}")
        response = requests.get(url, timeout=30)
        response.raise_for_status()
        return response.json()["data"]

    def transform(self, data):
        self.logger.info("Flattening nested API response")
        flat = []
        for record in data:
            flat.append({
                "id": record["id"],
                "name": record["attributes"]["name"],
                "email": record["attributes"]["email"],
                "_loaded_at": datetime.now().isoformat(),
                "_source": "api"
            })
        return flat

    def validate(self, data):
        """Override: API-specific validation."""
        super().validate(data)   # Parent's "not empty" check
        # Additional: check for required fields
        required = {"id", "attributes"}
        for record in data:
            missing = required - set(record.keys())
            if missing:
                raise ValueError(f"API record missing fields: {missing}")


class FilePipeline(BasePipeline):
    """Pipeline that extracts from CSV/Parquet files."""

    def extract(self):
        filepath = self.config["filepath"]
        self.logger.info(f"Reading file: {filepath}")
        if filepath.endswith(".csv"):
            return read_csv(filepath)
        elif filepath.endswith(".parquet"):
            return read_parquet(filepath)
        raise ValueError(f"Unsupported file format: {filepath}")

    def transform(self, data):
        self.logger.info("Cleaning file data")
        # Strip whitespace, standardize dates, etc.
        return [clean_record(row) for row in data]

Polymorphic Pipeline Runner

# The runner does not know or care what type of pipeline it is running
# It just calls .run() on each one — polymorphism handles the rest

def run_all_pipelines(pipelines):
    """Run a list of pipelines. Each can be SQL, API, File, or any new type."""
    results = {"success": [], "failed": []}

    for pipeline in pipelines:
        try:
            pipeline.run()     # Polymorphism — each type runs differently
            results["success"].append(pipeline.name)
        except Exception as e:
            results["failed"].append({"name": pipeline.name, "error": str(e)})

    # Summary
    total = len(pipelines)
    success = len(results["success"])
    failed = len(results["failed"])
    print(f"
Pipeline run complete: {success}/{total} succeeded, {failed} failed")

    return results

# Three different pipeline types — runner treats them identically
pipelines = [
    SQLPipeline("customers", {"source_table": "dbo.customers", "connection_string": "..."}),
    APIPipeline("users", {"api_url": "https://api.example.com/users"}),
    FilePipeline("products", {"filepath": "/data/products.csv"}),
]

results = run_all_pipelines(pipelines)
# Pipeline 'customers' STARTED
# Extracting from SQL: dbo.customers
# ...
# Pipeline 'users' STARTED
# Extracting from API: https://api.example.com/users
# ...
# Pipeline 'products' STARTED
# Reading file: /data/products.csv
# ...
# Pipeline run complete: 3/3 succeeded, 0 failed

# Adding a new source type (e.g., Kafka) requires:
# 1. Create KafkaPipeline(BasePipeline) with extract() and transform()
# 2. Add it to the list
# 3. ZERO changes to run_all_pipelines — it already works

Validator Hierarchy

from abc import ABC, abstractmethod

class BaseValidator(ABC):
    """Abstract validator — enforces the validate() contract."""

    def __init__(self, table_name):
        self.table_name = table_name
        self.errors = []

    @abstractmethod
    def validate(self, data):
        """Must return True/False and populate self.errors."""
        pass

    @property
    def is_valid(self):
        return len(self.errors) == 0

class NotNullValidator(BaseValidator):
    def __init__(self, table_name, columns):
        super().__init__(table_name)
        self.columns = columns

    def validate(self, data):
        for col in self.columns:
            null_count = sum(1 for row in data if row.get(col) is None)
            if null_count > 0:
                self.errors.append(f"{col}: {null_count} nulls")
        return self.is_valid

class UniqueValidator(BaseValidator):
    def __init__(self, table_name, column):
        super().__init__(table_name)
        self.column = column

    def validate(self, data):
        values = [row.get(self.column) for row in data]
        dupes = len(values) - len(set(values))
        if dupes > 0:
            self.errors.append(f"{self.column}: {dupes} duplicates")
        return self.is_valid

class RangeValidator(BaseValidator):
    def __init__(self, table_name, column, min_val, max_val):
        super().__init__(table_name)
        self.column = column
        self.min_val = min_val
        self.max_val = max_val

    def validate(self, data):
        out_of_range = sum(
            1 for row in data 
            if row.get(self.column) is not None 
            and not (self.min_val <= row[self.column] <= self.max_val)
        )
        if out_of_range > 0:
            self.errors.append(f"{self.column}: {out_of_range} out of [{self.min_val}, {self.max_val}]")
        return self.is_valid


# Polymorphic validation — run ALL validators regardless of type
def validate_data(data, validators):
    """Run all validators. Each type checks something different."""
    all_valid = True
    for validator in validators:
        if not validator.validate(data):    # Polymorphism
            all_valid = False
            for error in validator.errors:
                print(f"  ❌ {validator.table_name}: {error}")
    return all_valid

validators = [
    NotNullValidator("customers", ["id", "name", "email"]),
    UniqueValidator("customers", "id"),
    RangeValidator("customers", "age", 0, 150),
]

is_valid = validate_data(customer_data, validators)

Composed Pipeline with Pluggable Components

# Combining inheritance (for the interface) with composition (for flexibility)

from abc import ABC, abstractmethod

class BaseExtractor(ABC):
    @abstractmethod
    def extract(self, source): pass

class BaseTransformer(ABC):
    @abstractmethod
    def transform(self, data): pass

class BaseLoader(ABC):
    @abstractmethod
    def load(self, data, target): pass

# Concrete implementations
class SQLExtractor(BaseExtractor):
    def __init__(self, connection_string):
        self.connection_string = connection_string
    def extract(self, source):
        return query_database(self.connection_string, f"SELECT * FROM {source}")

class S3Extractor(BaseExtractor):
    def __init__(self, bucket):
        self.bucket = bucket
    def extract(self, source):
        return read_s3(self.bucket, source)

class CleansingTransformer(BaseTransformer):
    def transform(self, data):
        return [clean(row) for row in data]

class DeltaLoader(BaseLoader):
    def load(self, data, target):
        write_delta(data, target)

class ParquetLoader(BaseLoader):
    def load(self, data, target):
        write_parquet(data, target)


# Composed pipeline — swap any component without changing the pipeline
class ComposedPipeline:
    """Pipeline assembled from pluggable components."""

    def __init__(self, name, extractor, transformer, loader):
        if not isinstance(extractor, BaseExtractor):
            raise TypeError("extractor must be a BaseExtractor")
        if not isinstance(transformer, BaseTransformer):
            raise TypeError("transformer must be a BaseTransformer")
        if not isinstance(loader, BaseLoader):
            raise TypeError("loader must be a BaseLoader")

        self.name = name
        self.extractor = extractor
        self.transformer = transformer
        self.loader = loader

    def run(self, source, target):
        data = self.extractor.extract(source)
        data = self.transformer.transform(data)
        self.loader.load(data, target)

# Azure SQL → cleanse → Delta Lake
pipeline_a = ComposedPipeline(
    "sql_to_delta",
    SQLExtractor("server.database.windows.net"),
    CleansingTransformer(),
    DeltaLoader()
)

# S3 → cleanse → Parquet
pipeline_b = ComposedPipeline(
    "s3_to_parquet",
    S3Extractor("my-data-bucket"),
    CleansingTransformer(),
    ParquetLoader()
)

# Same transformer, different sources and targets — full flexibility

Common Mistakes

1. Forgetting super().__init__() — the parent’s attributes are never initialized. The child object is missing fundamental attributes, causing AttributeError at unpredictable times. Always call super().__init__() in the child’s __init__.
2. Inheriting when you should compose — “a Report is not a Database” but developers inherit from Database to reuse the query method. Ask “is X truly a type of Y?” If not, compose instead. Inheritance is the tightest coupling in OOP — use it only when the relationship is real.
3. Deep inheritance hierarchies — 4+ levels of inheritance become unreadable and fragile. A change in the root class can cascade unpredictably through all descendants. Keep hierarchies to 2-3 levels maximum. If you need more, switch to composition or mixins.
4. Not implementing all abstract methods — Python catches this at instantiation time with a TypeError, but developers sometimes miss the error in testing. Use IDE hints or type checkers (mypy) to catch missing implementations during development.
5. Using isinstance() excessively — if you are writing if isinstance(source, SQLSource): ... elif isinstance(source, APISource): ..., you are fighting polymorphism instead of using it. Let each class define its own behavior via method overriding.
6. Mixin classes with __init__ — if a mixin has an __init__, it competes with the main class’s __init__ in MRO. Keep mixins lightweight — use properties or lazy initialization instead of __init__.
7. Overriding without calling super() — when overriding a method that should extend (not replace) parent behavior, forgetting super().method() silently discards the parent logic. If the parent’s validate() checks for empty data and the child skips super(), the empty-data check is lost.

Interview Questions

Q: What is inheritance and why is it useful? A: Inheritance lets a child class reuse all attributes and methods of a parent class without rewriting them. The child can add new features or override existing ones. It eliminates code duplication — shared logic lives in the parent, and each child only contains what is unique to it. For example, SQLSource and APISource both inherit connection management from DataSource but implement extract() differently.

Q: What is the difference between method overriding and method overloading? A: Method overriding is when a child class redefines a method inherited from the parent — same name, same parameters, different behavior. Python uses overriding extensively. Method overloading (same name, different parameter types/counts) does not exist in Python — you use default parameters, *args/**kwargs, or singledispatch instead. Overriding changes behavior per type; overloading changes behavior per argument.

Q: What is super() and when do you use it? A: super() returns a reference to the parent class, letting you call its methods from the child. The most common use is super().__init__() to initialize parent attributes in the child’s constructor. You also use it when overriding a method and wanting to extend (not replace) parent behavior — call super().validate() to run the parent’s validation, then add your own checks.

Q: What is polymorphism? Give an example. A: Polymorphism means calling the same method on different types and getting type-specific behavior. Example: source.extract() called on a SQLSource runs a SQL query, on an APISource makes an HTTP request, and on a FileSource reads a CSV. The caller does not know or care which type — it just calls extract(). This enables writing generic functions like run_all_pipelines() that work with any source type.

Q: What is an abstract class and how is it different from a regular class? A: An abstract class (from the abc module) cannot be instantiated directly and can contain abstract methods that subclasses MUST implement. It defines a contract. A regular class can be instantiated and all its methods have implementations. Abstract classes catch missing method implementations at creation time (TypeError), not at runtime — preventing bugs where a pipeline fails mid-run because extract() was never defined.

Q: What is the difference between inheritance and composition? When do you use each? A: Inheritance models “is-a” relationships (SQLSource is a DataSource). Composition models “has-a” relationships (Pipeline has an Extractor, a Transformer, and a Loader). Inheritance shares implementation; composition shares interfaces. Use inheritance when the type relationship is real and the hierarchy is shallow (2-3 levels). Use composition when you want pluggable, swappable components — it is more flexible, more testable, and less brittle. The rule of thumb: if you are unsure, compose.

Q: What is the Method Resolution Order (MRO) and why does it matter? A: MRO is the order Python searches for methods in a class hierarchy, especially with multiple inheritance. It uses C3 linearization: child first, then parents left to right, each class exactly once. MRO matters because it determines which version of a method super() calls — in the diamond problem, MRO ensures each class’s method is called exactly once instead of duplicating calls to the shared grandparent.

Q: What are mixins and how are they different from regular inheritance? A: Mixins are small, single-purpose classes designed to add a specific capability (logging, serialization, timestamps) to other classes via multiple inheritance. Unlike regular base classes, mixins are not meant to be instantiated alone and do not represent a type hierarchy. They add features orthogonal to the class’s main purpose — a LoggingMixin adds logging to both Pipeline and Validator without those classes being related.

Wrapping Up

Advanced OOP gives you the tools to build scalable, maintainable pipeline systems. Inheritance lets you write shared logic once and specialize per source type. Method overriding lets each type customize behavior. Polymorphism lets your pipeline runner work with any source type without type-checking. Abstract classes enforce contracts so nothing slips through. Mixins add cross-cutting features like logging and serialization. And composition gives you the flexibility to assemble pipelines from pluggable, testable components.

The key insight is knowing when to use each tool. Inherit when the “is-a” relationship is real. Compose when you want flexibility. Use abstract classes to enforce contracts. Use mixins for cross-cutting concerns. And keep your hierarchies shallow — deep trees are where good code goes to die.

Previous in this series: OOP — Classes, __init__, self, Methods, and Properties

Next in this series: Modules, Packages, Virtual Environments, and Imports

---

Naveen Vuppula is a Senior Data Engineering Consultant and app developer based in Ontario, Canada. He writes about Python, SQL, AWS, Azure, and everything data engineering at DriveDataScience.com.