What is Data Engineering? Follow One Credit Card Swipe from Tap to Dashboard

Table of Contents

• One Swipe, Four Moments
• What Is Data Engineering?
• Bronze, Silver, and Gold — The Stages Data Goes Through
• Why Companies Need Data Engineering
• Batch vs Streaming
• ETL vs ELT
• Who Does What on a Data Team
• Follow the Swipe — A Real Architecture, Step by Step
• Watching One Record Travel Through the Platform
• Meet Maple Card — The Project We Will Build
• The Series Map — Build It, Run It, Migrate It
• Skills Roadmap for Beginners
• Common Misconceptions About Data Engineering
• Interview Questions
• What’s Next

This is the written companion to Episode 0 of the DriveDataScience YouTube series, where we build a complete data platform for a fictional credit card company, Maple Card, one episode at a time. Every episode has a written version like this one, with more detail than fits in a 10-minute video.

Every time you tap a credit card, a surprising amount happens behind the scenes — and most of it is data engineering. In this post we follow a single card swipe from the moment you tap, through a fraud check, onto your statement, and finally onto a dashboard at the bank. Along the way you will learn what data engineering actually is, why every modern company depends on it, the big design choices data engineers make every day, and how the pieces fit together in a real cloud architecture.

No experience is needed. If you have ever wondered what a data engineer does all day, this is the place to start.

One Swipe, Four Moments

Picture this. You buy a coffee and tap your card.

Moment 1 -- 0 seconds       You tap your card at the coffee shop.

Moment 2 -- ~2 seconds      The payment is approved. In those two seconds a
                            system checked: is this really you? Is this card
                            being used in another city right now?

Moment 3 -- next morning    The coffee appears on your card statement.

Moment 4 -- Monday          Someone at the bank opens a dashboard showing how
                            much customers spent on coffee last week.

None of that happens by itself. Someone designed and built the systems that carry that one swipe from the card machine to the fraud check, to your statement, to the dashboard — reliably, millions of times a day, without losing or double-counting a single transaction.

That someone is a data engineer.

What Is Data Engineering?

Here is the simple definition:

Data engineering is building the systems that move data from where it is created to where it is useful — and making sure it is clean, complete, and trustworthy when it arrives.

Analogy — City water. The best way to understand data engineering is to think about the water that comes out of your kitchen tap.

Notice that nobody at the tap thinks about the reservoir. They just expect clean water every time. That is exactly how good data engineering feels to the rest of a company: analysts open a dashboard and expect the numbers to be right, without ever thinking about the dozens of systems and pipelines behind them.

And just like a water utility, data engineering is mostly invisible when it works — and everyone notices immediately when it breaks.

Bronze, Silver, and Gold — The Stages Data Goes Through

Water goes through stages: raw from the reservoir, filtered at the plant, and finally safe to drink. Data goes through stages too. Most modern data platforms organize them into three layers, an approach called the medallion architecture:

Analogy — A restaurant kitchen. Bronze is the delivery of raw ingredients at the back door: muddy potatoes, whole fish, unwashed lettuce. Silver is the prep station: everything washed, peeled, and portioned. Gold is the plated dish the customer actually eats. You never throw away the delivery receipt (Bronze), because if a dish goes wrong you need to trace it back to the ingredients.

You will hear the words Bronze, Silver, and Gold in almost every episode of this series. Our medallion architecture deep dive covers the design in detail.

Why Companies Need Data Engineering

Imagine a credit card company with dozens of separate systems: one for transactions, one for customers, one for merchants, one for disputes, one for rewards points. Here is what life looks like with and without data engineering.

Analogy — A library with no catalogue. A library can own a million books, but without a catalogue, shelving system, and librarians, finding the one you need is nearly impossible. Data engineering is the catalogue and the shelving system for a company’s data. The books (the data) were always there; data engineering makes them findable, organized, and usable.

The business impact is real: faster decisions, fewer arguments about whose numbers are right, regulatory reports that can be produced on demand, and machine learning models that can only exist because clean, historical data is available to train them.

Batch vs Streaming

The first big design decision a data engineer makes for any data source is how often it needs to move.

BATCH
  What:     Collect data and process it on a schedule -- every hour, every night
  Example:  Your monthly card statement. Nobody needs it to the second.
  Pros:     Simpler to build, cheaper to run, easier to fix and rerun
  Cons:     Data is only as fresh as the last run

STREAMING
  What:     Process each event as it arrives, within seconds
  Example:  The fraud check on your swipe. If you wait until tonight,
            the fraudster is long gone.
  Pros:     Near real-time insight and action
  Cons:     More complex, more expensive, harder to debug and replay

Analogy — Mail vs a phone call. Batch is the postal service: letters are collected all day and delivered together the next morning. Efficient, reliable, and perfectly fine for most things. Streaming is a phone call: the message arrives the instant it is spoken. You would not phone someone to send them a monthly newsletter, and you would not mail a letter to warn them their house is on fire.

Most real companies use both. In the Maple Card project, most data — customers, merchants, chargebacks — loads in batch, while the live card swipes stream in real time so the fraud model can see them within seconds. Our Databricks structured streaming post shows the streaming side in action.

ETL vs ELT

The second big decision is where the data gets transformed. Both approaches use the same three steps — Extract, Transform, Load — in a different order.

ETL:   Extract  →  Transform  →  Load
       Clean and reshape the data BEFORE it reaches the warehouse.
       The norm for decades, when storage and compute were expensive and you
       only wanted to keep the finished product.

ELT:   Extract  →  Load  →  Transform
       Load the raw data FIRST, then transform it inside the platform.
       The standard approach in the cloud, where storage is cheap and compute
       is powerful and scalable.

Analogy — Cooking at home vs meal-kit delivery. ETL is buying groceries, cooking the meal at the store, and carrying only the finished dish home — if you want it seasoned differently tomorrow, you have to go back to the store. ELT is bringing all the raw ingredients home first and cooking in your own kitchen. Tomorrow you can cook something completely different from the same ingredients.

That flexibility is exactly why the Maple Card platform keeps a Bronze layer of untouched raw data: if a business rule changes six months from now, we can rebuild Silver and Gold from Bronze without asking the source systems for the data again.

Who Does What on a Data Team

“Data” jobs are easy to confuse. Here is how the main roles fit together.

Analogy — A highway system. The data engineer builds and maintains the roads: the pipelines that move and clean data. The data analyst drives on those roads, using them to reach answers. The data scientist or ML engineer builds the vehicles: models that make predictions. And the data platform engineer looks after the ground underneath: the land, the permits, the traffic rules — capacity, workspaces, access, and security.

In smaller companies, one person may wear several of these hats. In larger ones, each is a separate team. This series teaches mainly the data engineer and data platform engineer skills — with a season on AI at the end, because modern data engineers increasingly build the pipelines that feed machine learning.

Follow the Swipe — A Real Architecture, Step by Step

Now let’s put it all together. Here is the path our coffee swipe takes through a modern cloud data platform — the same platform we will build in this series.

1. SOURCE        2. INGEST           3. STORE          4. PROCESS          5. SERVE            6. USE
 Card machine  →  Synapse pipelines → Data lake      →  Databricks       →  Microsoft       →  Power BI
 Customer DB      (batch)             (ADLS Gen2)       Silver + Gold       Fabric              dashboards,
 Merchant list    Event Hubs          Bronze layer      tables                                  fraud model
                  (streaming)

A few things about this architecture are worth noticing early:

• Two paths in, one platform. Batch and streaming data arrive differently but end up in the same lake, cleaned by the same rules.
• Raw is always kept. Bronze is never edited, so every number in a dashboard can be traced back to the original swipe.
• Each tool does one job well. Synapse (and later Fabric) orchestrates and serves; Databricks does the heavy transformation. Many enterprises run exactly this combination.
• Security runs through every step. Card numbers are sensitive, so access is controlled at every layer — a theme we will return to many times.

Watching One Record Travel Through the Platform

Architecture diagrams can feel abstract, so let’s watch the actual data change as it moves through the layers. Here is our coffee swipe at each stage.

Bronze — exactly as the card machine sent it (note the duplicate: the terminal retried after a network hiccup):

{"txn_id": "T-88213", "card": "4519********0042", "amt": "4.75", "currency": "CAD",
 "merchant": "MAPLE BEAN CAFE #12 ", "ts": "2026-11-11T08:14:03-05:00", "status": "APPROVED"}
{"txn_id": "T-88213", "card": "4519********0042", "amt": "4.75", "currency": "CAD",
 "merchant": "MAPLE BEAN CAFE #12 ", "ts": "2026-11-11T08:14:03-05:00", "status": "APPROVED"}

Silver — cleaned, typed, deduplicated, and joined to reference data:

txn_id    customer_id  amount   currency  merchant_name       merchant_category  txn_timestamp_utc     is_duplicate_removed
T-88213   C-10442      4.75     CAD       Maple Bean Cafe      Coffee Shops       2026-11-11 13:14:03   yes (1 copy dropped)

What changed: the duplicate was removed, the amount became a number instead of text, the merchant name was trimmed and standardized, the merchant was matched to its category, the card was linked to a customer, and the timestamp was converted to UTC so transactions from every time zone line up.

Gold — aggregated to answer a business question (“How much did customers spend on coffee last week?”):

week_start   merchant_category  transactions  total_spend  avg_ticket  pct_change_vs_last_week
2026-11-09   Coffee Shops       184,302       $912,295     $4.95       +3.1%

That Gold row is what appears on Monday’s dashboard. The analyst who reads it never sees the duplicate swipe, the text-typed amount, or the messy merchant name — because a data engineer dealt with all of that upstream. (These numbers are illustrative; in the series we generate realistic data ourselves with Python.)

Analogy — Tracking a parcel. Every stage above is like a scan on a delivery tracking page: picked up, sorted, out for delivery, delivered. If a customer disputes a number on the dashboard, you can follow the tracking history all the way back to the original swipe in Bronze.

Meet Maple Card — The Project We Will Build

Most tutorials teach tools with toy datasets: a 50-row CSV of fruit sales, or the same Titanic file everyone has seen. Real data engineering does not look like that. So in this series, we build one realistic platform from start to finish for one company.

Maple Card is a fictional Canadian credit card company. The company is made up, but its data will be realistic: we generate customers, merchants, transactions, and chargebacks ourselves with Python, including the messy parts real systems produce — duplicates, late-arriving records, and a small percentage of fraudulent transactions.

Maple Card has four business problems, and our platform will solve all four:

The Series Map — Build It, Run It, Migrate It

Most tutorials stop once something is built. In real jobs, that is where the work begins: platforms must be run, secured, and eventually migrated to newer technology. This series covers all three.

How the series works:

• 8 to 10 minutes per episode, one idea each. Short enough to finish, focused enough to remember.
• Four questions every time: What is it? Why do we need it? When and where do we use it? How do we build it?
• Every episode adds one real piece to the Maple Card project, with the code on GitHub so you can join at any point.
• Every episode has a written version on drivedatascience.com, like this one.

Skills Roadmap for Beginners

If you want to become a data engineer, here is the order that works best — and it is exactly the order this series follows.

1. SQL          The language of data. Every data role uses it daily.
2. Python       Automation, data generation, APIs, and pipeline logic.
3. Cloud        Storage, compute, identity, and security in Azure.
4. Orchestration  Pipelines that move data on a schedule (Synapse, Data Factory, Fabric).
5. Spark        Processing data too big for one machine (PySpark, Databricks).
6. Modeling     Star schemas, Bronze/Silver/Gold, and serving data to BI.
7. Operations   CI/CD, monitoring, governance, and cost control.

You do not need to master each step before starting the next — but SQL and Python first will make everything after them much easier. If you prefer reading ahead, these written guides are a good starting point:

• Python for Data Engineers: The Essential Skills
• SQL Execution Order and Filtering
• Cloud Computing Fundamentals
• How Real Companies Receive Data

Common Misconceptions About Data Engineering

1. “Data engineering is just moving files around.” Moving data is the easy part. The real work is making it correct, complete, secure, and on time, every single day — handling duplicates, late data, schema changes, failures, and reruns without anyone downstream noticing.

2. “You need to be a math expert.” Data engineering is closer to software engineering than to statistics. Strong SQL, solid Python, and good problem-solving matter far more than advanced math.

3. “Real-time is always better.” Streaming is more complex and more expensive. Most business questions are answered perfectly well by data that refreshes hourly or nightly. Choose streaming only when acting late has a real cost, like fraud.

4. “Once a pipeline is built, the job is done.” Pipelines need monitoring, tuning, and maintenance. Source systems change, data volumes grow, and platforms get replaced — which is why this series includes a full migration season.

5. “Learning one tool is enough.” Tools change every few years; concepts do not. Batch vs streaming, ETL vs ELT, Bronze/Silver/Gold, data quality, and incremental loading apply on every platform. Learn the concepts, and each new tool becomes a variation on what you already know.

6. “Data engineers and data scientists do the same job.” They work closely together, but engineers build the reliable data foundation and scientists build models on top of it. A fraud model is only as good as the data pipeline that feeds it.

Interview Questions

Q: What does a data engineer do? A: A data engineer designs, builds, and maintains the systems that move data from where it is created to where it is used — ingesting it from source systems, storing it, cleaning and transforming it, and serving it to analysts, applications, and machine learning models. The goal is data that is reliable, timely, secure, and trusted across the company.

Q: What is the difference between batch and streaming processing? A: Batch processing collects data and processes it on a schedule, such as hourly or nightly; it is simpler and cheaper and suits reports like monthly statements. Streaming processes each event within seconds of it arriving; it suits use cases where acting late has a real cost, such as fraud detection. Most companies use both.

Q: Explain ETL vs ELT. A: Both extract, transform, and load data, in a different order. ETL transforms data before loading it into the warehouse, which was common when storage was expensive. ELT loads raw data first and transforms it inside the platform, which is standard in the cloud because storage is cheap and compute is scalable. ELT keeps the raw data, so you can reprocess it when business rules change.

Q: What is the medallion architecture? A: It organizes a data platform into three layers: Bronze holds raw data exactly as received, Silver holds cleaned, validated, and deduplicated data, and Gold holds aggregated, business-ready data for dashboards and models. It makes data quality improve step by step and lets every number be traced back to its source.

Q: How is a data engineer different from a data analyst and a data scientist? A: The data engineer builds the pipelines and platform that deliver reliable data. The data analyst uses that data to explain what happened, typically with SQL and BI tools. The data scientist uses it to predict what will happen, typically with Python and machine learning. A data platform engineer manages the underlying platform’s capacity, workspaces, access, and security.

Q: Why keep raw data if you already have clean data? A: Raw data is the source of truth. It lets you audit where a number came from, recover from bugs in transformation logic, and rebuild cleaned and aggregated tables when business rules change — without asking source systems to resend data that may no longer exist there.

What’s Next

That is data engineering in one sentence: building the pipes that take one swipe from a card machine to a fraud check, a statement, and a dashboard — and making sure the data is right at every step.

In Episode 1, we start building. We set up Python and VS Code, create a virtual environment, and scaffold the Maple Card project repository that will hold everything we build in this series. Subscribe to the DriveDataScience YouTube channel so you don’t miss it — and tell me in the comments where you are in your data journey.

Related posts: – The Medallion Architecture – How Real Companies Receive Data – Cloud Computing Fundamentals – Python for Data Engineers – Data Engineering Interview Questions