Neural Networks and Deep Learning Explained: Neurons, Backpropagation, Activation Functions, CNNs, RNNs, and Transformers

Table of Contents

• Why Neural Networks Matter for Data Engineers
• When Classical Machine Learning Is Not Enough
• The Artificial Neuron — One Tiny Decision Maker
• Activation Functions — Adding Non-Linearity
• Layers — From One Neuron to a Network
• The Forward Pass — Making a Prediction
• Loss Functions — Measuring How Wrong You Are
• Backpropagation and Gradient Descent — How Networks Learn
• Optimizers — Smarter Ways to Walk Downhill
• The Training Loop — Epochs, Batches, and Iterations
• Overfitting and Regularization
• Vanishing and Exploding Gradients
• Your First Neural Network in Python
• Convolutional Neural Networks (CNNs) — Seeing Images
• Recurrent Neural Networks (RNNs and LSTMs) — Remembering Sequences
• Transformers and Attention — The Architecture Behind LLMs
• Embeddings — Turning Meaning into Numbers
• Deep Learning in a Data Engineer’s Daily Work
• Classical ML vs Deep Learning — Decision Guide
• Common Mistakes
• Interview Questions
• Wrapping Up

So far in this series we have covered classical machine learning: linear and logistic regression, decision trees and random forests, XGBoost, and feature engineering. Those algorithms power most business models in production today. But every large language model, image classifier, speech recognizer, and embedding model you use is built on something else: neural networks.

This post explains neural networks and deep learning from the ground up — what a neuron actually computes, how a network learns from its mistakes, why some architectures see images and others read text, and how the transformer became the foundation of ChatGPT, Claude, and every model in the Azure AI Foundry catalog. Every number in the worked examples is calculated, so you can follow along with a calculator.

Analogy — A company learning to approve loans. Imagine a new company with hundreds of junior analysts (neurons), organized into departments (layers). Each analyst looks at the numbers handed to them, weighs some numbers as important and others as unimportant, and passes a single opinion up to the next department. The final department gives the verdict: approve or reject. On day one, everyone’s judgment is random, so the verdicts are terrible. But every time a verdict turns out wrong, the CEO traces the blame backwards through the company — the final department was 40% responsible, the analysts who fed them were partly responsible — and every analyst adjusts how much weight they give each input. After thousands of loan files, the company becomes remarkably good at approving loans, and nobody ever wrote down an explicit rule. That is a neural network: weights, layers, a verdict, and blame flowing backwards. The blame-tracing step is called backpropagation.

Why Neural Networks Matter for Data Engineers

You may never design a novel network architecture, but you will:

  1. FEED THEM DATA
     Deep learning is data-hungry. Training sets of millions of images,
     documents, or events come from YOUR pipelines -- Delta tables,
     feature stores, labeled datasets in ADLS or S3.

  2. RUN THEM AT SCALE
     GPU clusters in Databricks, Fabric Spark with ML runtimes,
     batch inference over billions of rows, real-time serving endpoints.

  3. USE THEIR OUTPUTS
     Embeddings for vector search (RAG), sentiment scores, extracted
     entities, document classifications -- all produced by neural networks
     and landed in tables you own.

  4. DEBUG THEM IN PRODUCTION
     "The model got worse" is often a data problem: schema drift,
     unnormalized features, leaked labels, skewed training data.
     Understanding how networks learn tells you what to check.

  5. ANSWER INTERVIEW QUESTIONS
     "What is backpropagation?" "Why ReLU over sigmoid?" "What is a
     transformer?" appear in data engineering and ML engineering interviews.

When Classical Machine Learning Is Not Enough

Classical algorithms like logistic regression and XGBoost need you to engineer the features. For a churn model that is fine: you create days_since_last_login, support_tickets_last_30d, and plan_type, and the algorithm finds the pattern.

Now try that with a photo. Which column describes “this image contains a cracked pipe”? There is no column — just 150,000 pixel values. The same is true for raw text, audio waveforms, and video. Hand-crafting features for this kind of data is practically impossible.

Neural networks learn the features themselves. Early layers discover simple patterns (edges in an image, character combinations in text), middle layers combine them into parts (corners, textures, words), and deep layers combine those into concepts (a cracked pipe, a complaint about billing). This is called representation learning, and it is the single biggest reason deep learning took over vision, speech, and language.

Data type            Classical ML              Deep learning
─────────            ────────────              ─────────────
Tabular (rows/cols)  Excellent (XGBoost wins)  Rarely better, more work
Images               Weak without features     State of the art (CNNs, ViTs)
Free text            OK with TF-IDF            State of the art (transformers)
Audio / speech       Weak                      State of the art
Time series          Strong (with lag features) Competitive for very long sequences

The Artificial Neuron — One Tiny Decision Maker

A neuron does three things:

1. Multiplies each input by a weight (how important that input is)
2. Adds the results together plus a bias (its baseline tendency)
3. Passes the sum through an activation function (decides how strongly to “fire”)

Formula:
  z      = (w1 × x1) + (w2 × x2) + ... + (wn × xn) + b
  output = activation(z)

Worked example -- a loan-approval neuron with two scaled inputs:

  x1 = 0.8   (income score, higher is better)
  x2 = 0.3   (debt ratio, higher is worse)

  w1 = 1.5   (income matters a lot, positively)
  w2 = -2.0  (debt matters a lot, negatively)
  b  = 0.1   (slight default leaning toward approval)

  z = (1.5 × 0.8) + (-2.0 × 0.3) + 0.1
    = 1.2 - 0.6 + 0.1
    = 0.7

  output = sigmoid(0.7) = 0.668  → 66.8% approval confidence

Analogy — A judge on a talent show. Each judge (neuron) scores contestants on several criteria. A singing judge weighs vocal range heavily (large weight) and costume lightly (small weight). They have a personal baseline — some judges are harsh, some generous (the bias). Their final score is a weighted sum of what they saw, adjusted by their baseline. The weights and bias are exactly what training changes.

If you recognize this formula, you should: a single neuron with a sigmoid activation is logistic regression. Neural networks become powerful when you connect many of these together in layers.

Activation Functions — Adding Non-Linearity

Without an activation function, stacking layers is pointless: a chain of weighted sums is still just one big weighted sum — a straight line, no matter how many layers you add. Activation functions bend the line, letting the network model curves, thresholds, and complex boundaries.

Same inputs, different activations:

  z      ReLU    Tanh     Sigmoid
  ─────  ─────   ──────   ───────
  -2     0       -0.964   0.119
   0     0        0.000   0.500
   2     2        0.964   0.881

Softmax example -- classifying a support ticket into 3 categories:

  Raw scores (logits):  billing = 2.0, technical = 1.0, account = 0.1
  Softmax output:       billing = 0.659, technical = 0.242, account = 0.099
                        (65.9% + 24.2% + 9.9% = 100%)

Why ReLU won. Sigmoid and tanh flatten out at the extremes, so their gradients become tiny and learning stalls in deep networks (see vanishing gradients below). ReLU’s gradient is exactly 1 for every positive input, it is extremely cheap to compute, and it made training deep networks practical around 2012.

Layers — From One Neuron to a Network

A fully connected (dense) network for the churn example:

  Input layer      Hidden layer 1    Hidden layer 2    Output layer
  (20 features) →  (64 neurons)   →  (32 neurons)   →  (1 neuron)
                   ReLU              ReLU              Sigmoid → churn probability

  Input layer:    receives the raw features (no computation)
  Hidden layers:  learn intermediate representations
  Output layer:   produces the prediction in the shape you need

Counting parameters (every connection is a weight, every neuron has a bias):

  Layer 1: 20 × 64 weights + 64 biases = 1,344
  Layer 2: 64 × 32 weights + 32 biases = 2,080
  Layer 3: 32 × 1  weights + 1  bias   =    33
  Total:                                  3,457 trainable parameters

For comparison:
  This small network      ~3.5 thousand parameters
  ResNet-50 (images)      ~25 million
  BERT-base (text)        ~110 million
  Modern LLMs             billions to trillions

“Deep” learning simply means many hidden layers. Width (neurons per layer) lets a layer capture more patterns at one level of abstraction; depth (number of layers) lets the network build abstractions on top of abstractions. Most of the progress in the last decade came from going deeper, with techniques that keep deep networks trainable.

The Forward Pass — Making a Prediction

The forward pass is simply data flowing from the input layer to the output layer, each layer applying “weights, sum, bias, activation” to the previous layer’s output.

Forward pass for one customer:

  Step 1: 20 feature values enter the input layer
  Step 2: each of the 64 neurons in hidden layer 1 computes its weighted
          sum of all 20 inputs, adds its bias, applies ReLU → 64 numbers
  Step 3: each of the 32 neurons in hidden layer 2 does the same with
          those 64 numbers → 32 numbers
  Step 4: the single output neuron combines the 32 numbers and applies
          sigmoid → 0.83 (83% probability this customer churns)

In practice this happens for a whole batch at once as matrix
multiplication: (batch × 20) · (20 × 64) → (batch × 64), and so on.
That is why GPUs, which multiply huge matrices in parallel, make deep
learning fast.

Loss Functions — Measuring How Wrong You Are

A loss function turns “how wrong was this prediction” into a single number the network can minimize.

Mean Squared Error (MSE) -- regression (predicting a number)
  loss = average of (prediction - actual)²
  Example: predicted delivery time 32 min, actual 30 min → error² = 4
  Squaring punishes big misses far more than small ones.

Binary Cross-Entropy -- yes/no classification
  loss = -[y × log(p) + (1 - y) × log(1 - p)]
  Customer actually churned (y = 1):
    predicted p = 0.9  → loss = 0.105  (confident and right: tiny loss)
    predicted p = 0.1  → loss = 2.303  (confident and wrong: huge loss)

Categorical Cross-Entropy -- multi-class classification
  Uses the softmax probability assigned to the correct class.
  Correct class "billing" got 0.659 → loss = -log(0.659) = 0.417

Rule of thumb:
  Predicting a number?           MSE (or MAE if outliers are common)
  Predicting yes/no?             Binary cross-entropy + sigmoid output
  Predicting one of N classes?   Categorical cross-entropy + softmax output

Analogy — A golf scorecard. The loss is your score for the round: lower is better, and every training step is about shaving strokes off. Cross-entropy is like a scorecard that adds a heavy penalty when you were confidently wrong — announcing “this ball is definitely going in” and then missing the green entirely costs far more than a hesitant miss.

Backpropagation and Gradient Descent — How Networks Learn

Training is a loop of four steps: predict, measure the error, assign blame, adjust.

Analogy — Walking down a mountain in fog. You cannot see the valley (the lowest loss), but you can feel the slope under your feet. Each step, you move a little in the steepest downhill direction. The slope is the gradient; the size of each step is the learning rate; walking downhill step by step is gradient descent. Backpropagation is the method for measuring the slope for every single weight in the network at once, by applying the chain rule from calculus backwards from the loss to the first layer — the CEO tracing blame through every department.

A one-weight example you can check by hand:

  Model:      prediction = w × x
  Data:       x = 2, actual y = 3
  Start:      w = 0.5
  Loss:       (prediction - y)²

  Step 1 -- forward:   prediction = 0.5 × 2 = 1.0
  Step 2 -- loss:      (1.0 - 3)² = 4.0
  Step 3 -- gradient:  d(loss)/dw = 2 × (prediction - y) × x
                                   = 2 × (1.0 - 3) × 2 = -8.0
                       Negative gradient → increasing w reduces the loss
  Step 4 -- update:    learning rate = 0.05
                       w_new = w - lr × gradient = 0.5 - 0.05 × (-8.0) = 0.9

  Check:  new prediction = 0.9 × 2 = 1.8
          new loss       = (1.8 - 3)² = 1.44   (down from 4.0)

Repeat thousands of times across millions of weights, and the network
converges toward weights that make good predictions.

The learning rate is the most important training setting:

  Too high (e.g., 1.0)      Steps overshoot the valley, loss bounces
                            around or explodes to NaN
  Too low (e.g., 0.000001)  Progress is painfully slow, training may
                            stall on a plateau
  Typical starting points   0.001 for Adam, 0.01–0.1 for SGD

Optimizers — Smarter Ways to Walk Downhill

Start with Adam (or AdamW) at a learning rate of 0.001. It works well out of the box for most problems, and you only need to revisit the choice if training is unstable or a published recipe recommends otherwise.

The Training Loop — Epochs, Batches, and Iterations

Dataset: 100,000 training rows, batch size 64

  Batch       A small group of rows processed together (64 rows).
              Gradients are averaged over the batch, which is smoother
              than one row at a time and fits in GPU memory.

  Iteration   One forward + backward pass + weight update for one batch.
              100,000 ÷ 64 = 1,563 iterations per epoch.

  Epoch       One complete pass through all 100,000 rows.
              Typical training: 10–100 epochs for small networks,
              often just 1–3 epochs when fine-tuning large models.

The three data splits:

  Training set     (70–80%)  Used to update the weights
  Validation set   (10–15%)  Checked after each epoch to tune settings
                             and decide when to stop
  Test set         (10–15%)  Touched ONCE at the very end for an honest
                             final score

Analogy — Studying for an exam. A batch is one study session on a handful of flashcards. An epoch is getting through the entire deck once. The validation set is a practice exam you take after each pass through the deck. The test set is the real exam — if you peek at it while studying, your score no longer means anything. (Our model evaluation guide covers splitting and cross-validation in depth.)

Overfitting and Regularization

Neural networks have so many parameters that they can memorize the training data instead of learning general patterns. You see it in the curves: training loss keeps falling while validation loss starts rising.

Epoch   Train loss   Validation loss
─────   ──────────   ───────────────
  1       0.62           0.60
  5       0.31           0.33
 10       0.18           0.27   ← best validation loss
 15       0.09           0.31   ← overfitting begins
 20       0.03           0.42   ← memorizing the training set

Vanishing and Exploding Gradients

Backpropagation multiplies gradients layer by layer. Multiply many numbers smaller than 1 and the result shrinks toward zero — the early layers barely learn (vanishing gradients). Multiply many numbers larger than 1 and the result explodes — weights jump wildly and loss becomes NaN (exploding gradients).

Analogy — A game of telephone. A message whispered through 50 people arrives either as a faint mumble (vanishing) or as a wildly exaggerated rumor (exploding). The people at the start of the chain never get useful feedback.

Fixes that made deep networks trainable:

  ReLU activations           Gradient of 1 for positive inputs, not ≤ 0.25
                             like sigmoid
  Careful initialization     He / Xavier initialization start weights at a
                             sensible scale
  Batch / layer normalization Keep each layer's values in a stable range
  Residual (skip) connections Let gradients flow around layers through
                             shortcuts -- the key idea in ResNet and every
                             transformer
  Gradient clipping          Cap the gradient size (common for RNNs and
                             transformers)
  LSTM / GRU gates           Purpose-built memory paths for sequences

Your First Neural Network in Python

A quick baseline with scikit-learn

scikit-learn’s MLPClassifier builds a small dense network in a few lines — perfect for experimenting on tabular data before reaching for a deep learning framework.

from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler
from sklearn.neural_network import MLPClassifier
from sklearn.linear_model import LogisticRegression

# Synthetic churn-style data: 5,000 customers, 20 features
X, y = make_classification(n_samples=5000, n_features=20, n_informative=10, random_state=42)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Neural networks need scaled inputs
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)

mlp = MLPClassifier(
    hidden_layer_sizes=(64, 32),   # two hidden layers
    activation="relu",
    solver="adam",
    alpha=1e-4,                    # L2 regularization
    early_stopping=True,           # holds out 10% for validation
    max_iter=200,
    random_state=42,
)
mlp.fit(X_train, y_train)
print("MLP accuracy:", round(mlp.score(X_test, y_test), 3))

baseline = LogisticRegression(max_iter=1000).fit(X_train, y_train)
print("Logistic regression accuracy:", round(baseline.score(X_test, y_test), 3))

# Output:
# MLP accuracy: 0.963
# Logistic regression accuracy: 0.852

The network beats logistic regression here because this synthetic dataset has non-linear interactions that a single straight decision boundary cannot capture. On many real tabular datasets, XGBoost would match or beat both — always compare against a strong classical baseline.

The same network in PyTorch

PyTorch is the most widely used deep learning framework, and it is available out of the box in Databricks ML runtimes and Fabric notebooks. Writing the training loop yourself shows every concept from this post in action.

import torch
from torch import nn
from torch.utils.data import DataLoader, TensorDataset

# Reuse the scaled arrays from the scikit-learn example
X_train_t = torch.tensor(X_train, dtype=torch.float32)
y_train_t = torch.tensor(y_train, dtype=torch.float32).unsqueeze(1)
X_test_t = torch.tensor(X_test, dtype=torch.float32)
y_test_t = torch.tensor(y_test, dtype=torch.float32).unsqueeze(1)

train_loader = DataLoader(TensorDataset(X_train_t, y_train_t), batch_size=64, shuffle=True)

# Architecture: 20 → 64 → 32 → 1   (3,457 parameters)
model = nn.Sequential(
    nn.Linear(20, 64),
    nn.ReLU(),
    nn.Dropout(0.2),        # regularization
    nn.Linear(64, 32),
    nn.ReLU(),
    nn.Linear(32, 1),       # raw score (logit); sigmoid is applied inside the loss
)

loss_fn = nn.BCEWithLogitsLoss()                      # sigmoid + binary cross-entropy, numerically stable
optimizer = torch.optim.Adam(model.parameters(), lr=1e-3)

for epoch in range(20):
    model.train()                                     # enables dropout
    for xb, yb in train_loader:
        optimizer.zero_grad()                         # clear old gradients
        loss = loss_fn(model(xb), yb)                 # forward pass + loss
        loss.backward()                               # backpropagation
        optimizer.step()                              # gradient descent update

    model.eval()                                      # disables dropout
    with torch.no_grad():
        test_loss = loss_fn(model(X_test_t), y_test_t).item()
    if (epoch + 1) % 5 == 0:
        print(f"epoch {epoch + 1:2d}  last batch loss {loss.item():.3f}  test loss {test_loss:.3f}")

model.eval()
with torch.no_grad():
    probs = torch.sigmoid(model(X_test_t))
    preds = (probs > 0.5).float()
    accuracy = (preds == y_test_t).float().mean().item()
print(f"PyTorch accuracy: {accuracy:.3f}")

Map the code to the concepts:

  nn.Linear + nn.ReLU            Neurons and activation functions
  nn.Dropout(0.2)                Regularization (20% of neurons off per step)
  BCEWithLogitsLoss              Binary cross-entropy loss
  DataLoader(batch_size=64)      Batches
  for epoch in range(20)         Epochs
  loss.backward()                Backpropagation
  optimizer.step()               Gradient descent (Adam)
  model.train() / model.eval()   Dropout on during training, off for prediction
  torch.no_grad()                No gradient tracking during evaluation (faster,
                                 less memory)

In a real project you would score a separate validation set each epoch to drive early stopping and keep the test set for the final check only. Exact numbers vary slightly between runs because of random initialization and shuffling.

Convolutional Neural Networks (CNNs) — Seeing Images

A dense network treats an image as a flat list of numbers and needs an enormous number of weights to do so.

Why dense layers fail on images:

  One 224 × 224 color image = 224 × 224 × 3 = 150,528 input values
  A dense layer to just 1,000 neurons = 150,528 × 1,000
                                      = 150,528,000 weights
  ...and it ignores that neighboring pixels belong together.

A convolution instead slides a small filter across the image:

  One 3 × 3 filter over 3 color channels, 32 filters:
      3 × 3 × 3 × 32 weights + 32 biases = 896 parameters
  The same 896 weights are reused at every position in the image.

Analogy — Inspecting a wall with a small flashlight. Instead of trying to take in the whole wall at once, you sweep a small flashlight across it, looking for one specific thing at a time — a crack, a stain, a rivet. Each filter is one flashlight looking for one pattern. Early filters find edges and color changes; deeper layers combine them into textures, then shapes, then objects. Pooling layers shrink the picture between steps, like stepping back from the wall to see larger regions.

Typical CNN pipeline:

  Image → [Conv → ReLU → Pool] × several → Flatten → Dense → Softmax

  Conv layers     Detect local patterns with shared filters
  Pooling         Downsample (e.g., keep the max of each 2×2 block)
  Dense head      Combine detected features into a final decision

Real-world uses:
  Manufacturing   Defect detection on assembly lines
  Healthcare      Medical image screening
  Retail          Shelf monitoring, product recognition
  Documents       OCR, layout detection (inside services like
                  Azure Document Intelligence)

Modern vision models also use Vision Transformers (ViTs), which apply the attention mechanism described below to image patches. Services like Azure AI Vision and Custom Vision use these architectures behind the scenes — see our computer vision post.

Recurrent Neural Networks (RNNs and LSTMs) — Remembering Sequences

Text, speech, sensor readings, and clickstreams are sequences: order matters. “Dog bites man” and “man bites dog” contain the same words with very different meanings.

Analogy — Reading a book one word at a time while keeping notes. An RNN reads one item at a time and carries a running summary — its hidden state — from step to step. Each new word updates the notes. The problem: by page 300, the notes from page 1 have been overwritten so many times they are nearly gone (vanishing gradients through time).

RNN family:

  Basic RNN    Hidden state carried forward; forgets long-range context
  LSTM         Adds gates: forget gate (what to erase), input gate (what
               to write), output gate (what to reveal). Remembers much
               longer contexts.
  GRU          A simpler LSTM with two gates; often similar accuracy,
               faster to train

Strengths:   Natural fit for streams; constant memory per step
Weaknesses:  Must process tokens one after another (slow to train,
             hard to parallelize on GPUs); still struggles with very
             long documents

Where you still see them:
  Smaller time-series forecasting models, streaming anomaly detection,
  on-device models with tight memory budgets

Transformers and Attention — The Architecture Behind LLMs

In 2017, the paper “Attention Is All You Need” introduced the transformer, which drops recurrence entirely. Instead of reading word by word, it looks at all words at once and lets every word decide which other words matter to it.

Analogy — A meeting instead of a game of telephone. An RNN is a telephone chain: information passes person to person and degrades. A transformer is a meeting where everyone is in the same room. When someone says “it,” every person can look around the table to see who “it” refers to. Each word “pays attention” to every other word, weighted by relevance.

Self-attention in one sentence:

  "The pipeline failed because it ran out of memory."

  When processing "it", attention weights might look like:
    pipeline  0.71   ← "it" mostly refers to the pipeline
    memory    0.12
    failed    0.09
    (others)  0.08

How attention works (Query, Key, Value):
  Each word produces three vectors:
    Query  "What am I looking for?"
    Key    "What do I contain?"
    Value  "What information do I pass along?"
  Relevance = how well a word's Query matches each other word's Key.
  Output   = a relevance-weighted blend of the Values.

A transformer block:
  Multi-head self-attention  (several attention patterns in parallel)
  → Add + LayerNorm          (residual connection + normalization)
  → Feed-forward network     (dense layers, often with GELU)
  → Add + LayerNorm
  Stack 12, 24, 96+ of these blocks → BERT, GPT, Claude, Llama

Why transformers took over:

  Parallel training    Every position is processed at once → GPUs are
                       used efficiently → models scale to billions of
                       parameters
  Long-range context   Any word can attend to any other word directly
  Transfer learning    Pretrain once on huge text corpora, then adapt
                       to your task (fine-tuning, prompting, RAG)

Three transformer families:
  Encoder-only    (BERT)       Understanding: classification, NER, embeddings
  Decoder-only    (GPT, Claude, Llama)  Generation: chat, code, summaries
  Encoder-decoder (T5)         Translation, summarization

This is the architecture behind every large language model you use. Our fine-tuning LLMs guide covers adapting these models to your domain, and the AI-103 series covers using them through prompt engineering and RAG pipelines.

Embeddings — Turning Meaning into Numbers

An embedding is the list of numbers a neural network produces internally to represent an item — a word, a sentence, a product, a customer. Items with similar meaning end up with similar numbers.

Analogy — GPS coordinates for meaning. Every sentence gets a location on a map with hundreds or thousands of dimensions. “Reset my password” and “I forgot my login” land close together even though they share no words; “quarterly revenue report” lands far away.

Where data engineers meet embeddings:

  Vector search / RAG    Embed document chunks, store vectors in Azure AI
                         Search, Databricks Vector Search, or pgvector;
                         embed the question and find the nearest chunks
  Deduplication          Find near-duplicate records with different wording
  Recommendations        "Customers who viewed this also viewed..."
  Clustering             Group support tickets by topic without labels
  Features for ML        Feed text embeddings into XGBoost as numeric columns

Typical sizes:
  text-embedding-3-small    1,536 dimensions
  text-embedding-3-large    3,072 dimensions
  Storage: 1M chunks × 1,536 floats × 4 bytes ≈ 6 GB of vectors

Deep Learning in a Data Engineer’s Daily Work

Training data pipelines
  Curate labeled datasets as versioned Delta tables (time travel lets you
  reproduce exactly which data trained which model). Track label quality,
  class balance, and leakage between train and test splits.

Compute
  Databricks ML runtimes with GPU clusters; distributed training with
  TorchDistributor or Ray on Databricks; Fabric notebooks with ML
  environments. GPUs are expensive -- size clusters for the job and shut
  them down after training.

Experiment tracking and registry
  MLflow (built into Databricks and Fabric) logs parameters, metrics,
  and model artifacts, and versions models in a registry so you can
  promote, compare, and roll back.

Inference at scale
  Batch: apply a model to millions of rows with Spark (pandas UDFs or
  mlflow.pyfunc.spark_udf) and write predictions to a Delta table.
  Real-time: model serving endpoints in Databricks or Azure AI Foundry.

Monitoring
  Watch input drift (feature distributions changing), prediction drift,
  and data quality on the inputs. Most "model got worse" incidents are
  pipeline changes, not model bugs.

Classical ML vs Deep Learning — Decision Guide

The practical rule: for tabular business data, start with XGBoost. For unstructured data, start with a pretrained model — an Azure AI service, an embedding model, or a fine-tuned foundation model — before considering training a network from scratch.

Common Mistakes

1. Using deep learning for small tabular datasets. A neural network on 2,000 rows of customer data usually loses to a well-tuned XGBoost model, takes longer to train, and is harder to explain. Neural networks shine on unstructured data and very large datasets. Always benchmark against a strong classical baseline first.

2. Forgetting to scale input features. Networks learn with gradient descent, so features on very different scales (age 0–100, income 0–500,000) make the loss surface lopsided and training slow or unstable. Standardize or normalize inputs, and apply the exact same scaler (fit on training data only) at inference time.

3. Choosing the wrong output activation and loss pair. Binary classification uses one output with sigmoid and binary cross-entropy; multi-class uses N outputs with softmax and categorical cross-entropy; regression uses a linear output with MSE. In PyTorch, BCEWithLogitsLoss and CrossEntropyLoss apply sigmoid or softmax internally — adding it again in the model is a classic bug.

4. Ignoring the validation curve. Training loss always goes down; that alone proves nothing. Plot validation loss every epoch, use early stopping, and keep the best checkpoint rather than the last one.

5. Picking a learning rate at random. Too high causes loss to explode to NaN; too low wastes hours of GPU time. Start with 0.001 for Adam, watch the first few hundred steps, and adjust by factors of 10.

6. Leaking the test set. Tuning architecture, epochs, or thresholds while looking at test-set results inflates your reported accuracy. The test set is used once, at the end.

7. Training from scratch when a pretrained model exists. For text, images, and speech, fine-tuning or prompting a pretrained model typically beats a from-scratch network trained on your limited data, at a fraction of the cost.

8. Leaving dropout on during prediction. Forgetting model.eval() in PyTorch keeps dropout active, so predictions become random and accuracy drops. Always switch to evaluation mode, and wrap inference in torch.no_grad().

Interview Questions

Q: What is a neural network, and how does a single neuron work? A: A neural network is layers of simple computing units (neurons) connected by weighted links. Each neuron multiplies its inputs by weights, adds a bias, and passes the sum through an activation function. Stacked in layers, these neurons learn increasingly abstract representations of the input. A single neuron with a sigmoid activation is mathematically identical to logistic regression.

Q: Why do neural networks need activation functions? A: Without a non-linear activation, any stack of layers collapses into a single linear transformation, so the network could only learn straight-line relationships no matter how deep it is. Activation functions like ReLU introduce non-linearity, letting networks model complex curves and decision boundaries. ReLU is the default for hidden layers because it is cheap to compute and avoids the vanishing gradients that sigmoid and tanh cause in deep networks.

Q: Explain backpropagation and gradient descent. A: Gradient descent minimizes the loss by repeatedly nudging each weight in the direction that reduces the loss, scaled by the learning rate. Backpropagation is the algorithm that computes those directions (gradients) efficiently for every weight, by applying the chain rule from the loss backwards through each layer. Training repeats forward pass, loss calculation, backpropagation, and weight update for every batch across many epochs.

Q: What is the difference between an epoch, a batch, and an iteration? A: A batch is a small subset of training examples processed together. An iteration is one forward pass, backward pass, and weight update on one batch. An epoch is one full pass through the entire training dataset. With 100,000 rows and a batch size of 64, one epoch is about 1,563 iterations.

Q: How do you detect and prevent overfitting in a neural network? A: Overfitting shows up as training loss continuing to fall while validation loss rises. Prevent it with early stopping on validation loss, dropout, weight decay, data augmentation, a smaller network, and above all more training data. Keep the checkpoint with the best validation loss rather than the final one.

Q: What is the vanishing gradient problem and how is it solved? A: Backpropagation multiplies gradients across layers, so with activations like sigmoid the gradients shrink toward zero in early layers and those layers stop learning. Solutions include ReLU activations, careful weight initialization, batch or layer normalization, residual connections that let gradients skip layers, and gated architectures like LSTMs for sequences.

Q: Compare CNNs, RNNs, and transformers. A: CNNs use small filters shared across an image to detect local patterns, making them efficient for images and spatial data. RNNs and LSTMs process sequences one step at a time while carrying a hidden state, suiting streams but training slowly and struggling with long contexts. Transformers use self-attention to let every token relate directly to every other token in parallel, which scales efficiently on GPUs and handles long context; they power modern LLMs and increasingly vision models too.

Q: When would you choose XGBoost over a neural network? A: For structured tabular data — customer records, transactions, sensor aggregates — XGBoost usually matches or beats neural networks with less data, less compute, faster iteration, and better explainability. Neural networks are the right choice for unstructured data like images, text, and audio, or when you can build on a pretrained model.

Wrapping Up

A neural network is a remarkably simple idea repeated at enormous scale: neurons compute weighted sums, activation functions add non-linearity, layers build abstractions on top of abstractions, a loss function measures the error, and backpropagation with gradient descent nudges millions of weights toward better predictions. CNNs adapt the idea for images, RNNs for sequences, and transformers — with self-attention — for the language models that now sit behind chat assistants, embeddings, and RAG systems.

For data engineers, the practical takeaway is that deep learning runs on data pipelines: clean, versioned, well-labeled training data, scalable GPU compute, tracked experiments, and monitored inference. Start with XGBoost for tabular problems, start with a pretrained model for unstructured ones, and use the concepts in this post to understand — and debug — what is happening inside.

Next, our fine-tuning LLMs guide takes the transformer architecture and shows how to adapt a pretrained language model to your own data.

Related posts: – AI/ML Introduction – Linear and Logistic Regression – Feature Engineering – Model Evaluation – Hyperparameter Tuning – Fine-Tuning LLMs – RAG Pipelines on Azure

Naveen Vuppula is a Senior Data Engineer and app developer based in Ontario, Canada. He writes about Python, SQL, AWS, Azure, and everything data engineering at DriveDataScience.com.