Databricks Notebooks Deep Dive for Data Engineers: Cell Types, Magic Commands, Widgets, Parameterization, dbutils.notebook.run, display and displayHTML, Notebook-Scoped Libraries, Collaboration, Modular Code, and Production Patterns

Table of Contents

• Notebook Basics
• Magic Commands — Switching Languages Mid-Notebook
• Widgets — Parameterized Notebooks
• display() and displayHTML() — Rich Output
• Notebook-Scoped Libraries
• Notebook Orchestration — %run and dbutils.notebook.run
• Collaboration Features
• Modular Notebook Design for Production
• Notebook Best Practices for Data Engineers
• Common Mistakes
• Interview Questions
• Wrapping Up

Our Databricks Intro & dbutils post introduced the Databricks workspace and core dbutils commands. This post goes deep into notebooks — the primary development environment where data engineers write, test, and deploy code. Notebooks look simple on the surface (just cells and a run button), but they have powerful features that most engineers never fully explore: magic commands that switch languages mid-notebook, widgets that turn notebooks into parameterized pipelines, orchestration commands that chain notebooks together, and collaboration features that let teams work on the same notebook simultaneously.

Analogy — A Swiss Army workbench. A Databricks notebook is not just a code editor — it is a complete workbench with multiple tools built in. The main blade is Python or SQL (your default language). The screwdriver attachment is magic commands (%sql, %scala, %r) that let you switch tools without leaving the bench. The measuring tape is widgets (parameterized inputs that make your notebook flexible). The clamp is %run and dbutils.notebook.run (they hold notebooks together into pipelines). The storage drawer is display() and displayHTML() (they show results in rich formats). And the label maker is %md (markdown cells that document everything). This post teaches you every tool on the workbench.

Notebook Basics

Creating and Configuring Notebooks

When you create a notebook in Databricks, you choose a default language (Python, SQL, Scala, or R) and attach it to a compute cluster. The default language determines what language runs when you do not specify a magic command at the top of a cell.

Creating a notebook:
  1. Workspace -> click your user folder or a shared folder
  2. Right-click -> Create -> Notebook
  3. Name: "orders_pipeline" (use descriptive names)
  4. Default Language: Python (most common for data engineering)
  5. Cluster: Select an existing cluster or create a new one

Notebook structure:
  - Each notebook is a sequence of cells
  - Cells can contain code, SQL, markdown, or shell commands
  - Cells execute independently but share the same SparkSession
  - Variables defined in one cell are available in all subsequent cells
  - Results display directly below each cell

Cell Types

Every cell in a Databricks notebook has a type determined by its first line. The default type matches the notebook’s default language. You switch types using magic commands.

Magic Commands — Switching Languages Mid-Notebook

Magic commands let you use multiple languages in the same notebook. A Python notebook can run SQL queries, execute shell commands, install packages, and render documentation — all without leaving the notebook.

Analogy — A multilingual conference call. The call (notebook) starts in English (Python), but when the French delegate needs to speak (SQL query), they press a button (%sql) and everyone hears French. When someone needs to check the building’s air conditioning (%sh), they call maintenance directly. Magic commands are the “language switch buttons” on the conference phone.

Language Magic Commands

# Default cell -- runs Python (no magic command needed)
df = spark.read.table("orders")
df.count()

%sql
-- Switch to SQL in any cell
SELECT region, COUNT(*) as order_count, SUM(amount) as revenue
FROM orders
GROUP BY region
ORDER BY revenue DESC

# Back to Python -- access SQL results using _sqldf
# The last SQL query result is automatically available as _sqldf
sql_result_df = _sqldf
display(sql_result_df)

%scala
// Switch to Scala for performance-critical operations
val df = spark.table("orders")
val count = df.count()
println(s"Total orders: $count")

%r
# Switch to R for statistical analysis
library(SparkR)
df <- sql("SELECT * FROM orders")
summary(df)

Markdown Magic Command

%md
# Orders Pipeline Documentation

Purpose:
This notebook processes daily vendor orders from the bronze layer,
cleans and validates them, and loads to the silver layer.

Parameters:
- date: Processing date (YYYY-MM-DD)
- region: Filter by region (default: all)

Schedule:
Runs daily at 6 AM via Databricks Workflows

| Step | Description | Output Table |
|------|-------------|-------------|
| 1    | Extract from bronze | bronze.orders |
| 2    | Clean and validate | silver.orders_clean |
| 3    | Aggregate | gold.revenue_by_region |

Shell Magic Command

%sh
# Run bash commands on the driver node (NOT on workers)
echo "Python version:"
python --version

echo "Disk space:"
df -h

echo "Current user:"
whoami

# Check if a file exists
ls -la /tmp/data/

File System Magic Command

%fs
# Shorthand for dbutils.fs commands
ls /Volumes/main/default/my-volume/

# Same as: dbutils.fs.ls("/Volumes/main/default/my-volume/")

Pip Magic Command

%pip install pandas==2.1.4 requests==2.31.0 pyarrow==14.0.2

# After %pip install, you MUST restart the Python interpreter
# Databricks shows a "Restart Python" button -- click it
# Or use: dbutils.library.restartPython()

Widgets — Parameterized Notebooks

Widgets add interactive inputs (text boxes, dropdowns, combo boxes, multi-selects) to your notebook. They appear in a panel at the top of the notebook and let you run the same notebook with different parameters without changing code.

Analogy — A vending machine with buttons. The vending machine (notebook) always does the same thing (process data), but the buttons (widgets) let you choose what it produces. Press “A1” (date = 2026-07-19) and you get today’s data. Press “B3” (region = Ontario) and you get Ontario’s data. The machine’s internal logic never changes — only the inputs.

Creating Widgets

# Text widget -- free-form text input
dbutils.widgets.text("processing_date", "2026-07-19", "Processing Date")

# Dropdown widget -- select one from a list
dbutils.widgets.dropdown("region", "all", ["all", "Ontario", "Quebec", "Alberta", "BC"], "Region")

# Combobox widget -- dropdown with free-text option
dbutils.widgets.combobox("output_format", "delta", ["delta", "parquet", "csv"], "Output Format")

# Multiselect widget -- select multiple values
dbutils.widgets.multiselect("categories", "Widget", ["Widget", "Gadget", "Doohickey", "All"], "Categories")

Using Widget Values

# Get widget values
date = dbutils.widgets.get("processing_date")
region = dbutils.widgets.get("region")
output_format = dbutils.widgets.get("output_format")
categories = dbutils.widgets.get("categories")  # Returns comma-separated string

print(f"Processing: date={date}, region={region}, format={output_format}")

# Use in queries
if region == "all":
    df = spark.sql(f"SELECT * FROM bronze.orders WHERE order_date = '{date}'")
else:
    df = spark.sql(f"SELECT * FROM bronze.orders WHERE order_date = '{date}' AND region = '{region}'")

%sql
-- Use widgets in SQL cells with :param_name syntax (Databricks Runtime 15.2+)
SELECT * FROM bronze.orders
WHERE order_date = :processing_date
  AND (region = :region OR :region = 'all')

Managing Widgets

# Remove a specific widget
dbutils.widgets.remove("processing_date")

# Remove ALL widgets (useful at the start of notebooks)
dbutils.widgets.removeAll()

Widget Behavior Settings

Click the gear icon next to the widget panel to configure:

  "Run Notebook"         -- Reruns entire notebook when a widget value changes
  "Run Accessed Commands" -- Reruns only cells that use the changed widget (default)
  "Do Nothing"           -- No automatic rerun (best for development)

For development: use "Do Nothing" to avoid accidental reruns.
For dashboards: use "Run Accessed Commands" for interactive exploration.

display() and displayHTML() — Rich Output

display() — Formatted Tables and Charts

display() is Databricks’ enhanced version of show(). It renders DataFrames as interactive tables with sorting, filtering, and built-in chart visualizations.

# Basic display -- interactive table with sorting and filtering
df = spark.read.table("orders")
display(df)

# display() vs show()
df.show()     # Plain text, fixed width, truncated -- good for debugging
display(df)   # Rich HTML table, sortable, filterable, chartable -- good for exploration

# display() also works with pandas DataFrames
import pandas as pd
pdf = pd.DataFrame({"product": ["Widget", "Gadget"], "revenue": [1000, 2000]})
display(pdf)

# After calling display(), click the chart icon below the table
# to create bar charts, line charts, pie charts, scatter plots, etc.
# No matplotlib or plotly needed for basic visualization

displayHTML() — Custom HTML Rendering

# Render custom HTML in a cell
displayHTML("<h2 style='color: green;'>Pipeline Completed Successfully!</h2>")

# Render a styled status card
displayHTML("""
<div style="padding: 20px; background: #0f172a; border-radius: 8px; color: white; max-width: 500px;">
  <h3 style="color: #60a5fa; margin-top: 0;">Pipeline Status</h3>
  <p>Rows processed: <strong>15,420</strong></p>
  <p>Duration: <strong>45 seconds</strong></p>
  <p>Status: <span style="color: #4ade80;">SUCCESS</span></p>
</div>
""")

# Render a progress indicator
def show_progress(step, total, message):
    pct = int(step / total * 100)
    displayHTML(f"""
    <div style="padding: 10px; max-width: 400px;">
      <p><strong>Step {step}/{total}:</strong> {message}</p>
      <div style="background: #e2e8f0; border-radius: 4px; overflow: hidden;">
        <div style="width: {pct}%; background: #60a5fa; height: 20px;"></div>
      </div>
      <p style="font-size: 12px; color: #94a3b8;">{pct}% complete</p>
    </div>
    """)

show_progress(2, 5, "Transforming data...")

Notebook-Scoped Libraries

Databricks supports installing Python packages at two levels: cluster libraries (available to all notebooks on the cluster) and notebook-scoped libraries (available only to the current notebook). For data engineering, notebook-scoped libraries are preferred because they avoid version conflicts between notebooks.

# Install packages for THIS notebook only
%pip install requests==2.31.0 beautifulsoup4==4.12.3

# After %pip install, restart the Python interpreter
dbutils.library.restartPython()

# Now use the installed packages
import requests
from bs4 import BeautifulSoup

# Check installed version
import importlib.metadata
print(importlib.metadata.version("requests"))

# Install from a requirements file
%pip install -r /Volumes/main/default/my-volume/requirements.txt

# Install a private package from a wheel file
%pip install /Volumes/main/default/my-volume/my_package-1.0.0-py3-none-any.whl

Analogy — Bringing your own tools to a shared workshop. Cluster libraries are the tools bolted to the workbench — everyone uses the same hammer and saw. Notebook-scoped libraries are the tools in your personal toolbox — you bring exactly what you need, and it does not affect anyone else’s setup.

Notebook Orchestration — %run and dbutils.notebook.run

Databricks provides two ways to call one notebook from another. Understanding the difference is critical for building modular, production-grade pipelines.

%run — Inline Execution (Same Context)

%run executes another notebook inline — as if its code were pasted into the current notebook. All variables, functions, and imports from the called notebook become available in the calling notebook. Think of it as an import statement.

# Run a shared utilities notebook
%run ./utils/common_functions

# After %run, all functions from common_functions are available
# If common_functions defined: def clean_columns(df): ...
cleaned = clean_columns(raw_df)

# Pass widget values to %run
%run ./process_orders $processing_date="2026-07-19" $region="Ontario"

%run behavior:
  - Runs synchronously (blocks until complete)
  - Shares the same SparkSession and variables
  - Functions and variables are available after %run
  - Cannot return a value (use shared variables instead)
  - Path is relative (./) or absolute (/Users/naveen/notebooks/)
  - The called notebook must exist at run time

Best for:
  - Shared utility functions
  - Configuration loading
  - Reusable transformation logic

dbutils.notebook.run — Job Execution (Separate Context)

dbutils.notebook.run() executes another notebook as a separate job. It runs in its own isolated context (separate variables, separate SparkSession). The called notebook can return a string value using dbutils.notebook.exit().

# Run a notebook and get its return value
result = dbutils.notebook.run(
    path="./process_orders",
    timeout_seconds=600,  # 10-minute timeout (0 = no timeout)
    arguments={"processing_date": "2026-07-19", "region": "Ontario"}
)
print(f"Result: {result}")
# Result: "Processed 15420 rows"

# The called notebook uses dbutils.notebook.exit() to return a value
# In process_orders notebook:
# dbutils.notebook.exit(f"Processed {row_count} rows")

# Orchestrate multiple notebooks sequentially
import json

steps = [
    {"path": "./01_extract", "args": {"source": "vendor_a"}},
    {"path": "./02_transform", "args": {"quality_threshold": "0.95"}},
    {"path": "./03_load", "args": {"target": "silver.orders"}},
]

results = {}
for step in steps:
    try:
        result = dbutils.notebook.run(
            path=step["path"],
            timeout_seconds=1800,
            arguments=step["args"]
        )
        results[step["path"]] = {"status": "success", "result": result}
        print(f"  {step['path']}: SUCCESS -- {result}")
    except Exception as e:
        results[step["path"]] = {"status": "failed", "error": str(e)}
        print(f"  {step['path']}: FAILED -- {e}")
        raise  # Stop pipeline on failure

print(f"\nPipeline complete: {json.dumps(results, indent=2)}")

# Run notebooks in parallel using concurrent.futures
from concurrent.futures import ThreadPoolExecutor
import json

def run_notebook(config):
    result = dbutils.notebook.run(
        path=config["path"],
        timeout_seconds=config.get("timeout", 1800),
        arguments=config.get("args", {})
    )
    return {"path": config["path"], "result": result}

notebooks = [
    {"path": "./extract_vendor_a", "args": {"date": "2026-07-19"}},
    {"path": "./extract_vendor_b", "args": {"date": "2026-07-19"}},
    {"path": "./extract_vendor_c", "args": {"date": "2026-07-19"}},
]

with ThreadPoolExecutor(max_workers=3) as executor:
    results = list(executor.map(run_notebook, notebooks))

for r in results:
    print(f"  {r['path']}: {r['result']}")

%run vs dbutils.notebook.run Comparison

Collaboration Features

Databricks notebooks support real-time collaboration — multiple engineers can edit the same notebook simultaneously, like Google Docs for code.

Real-time co-editing:
  - Multiple users can edit different cells simultaneously
  - Cursors show each user's position (color-coded)
  - Changes sync in real time

Comments:
  - Select code -> right-click -> "Comment"
  - Comments appear as threads in the margin
  - Team members can reply and resolve comments
  - Use for code review before promoting to production

Version history:
  - Every save creates a version
  - Access: File -> Revision History
  - Compare any two versions side by side
  - Restore a previous version with one click
  - Linked to Git via Databricks Repos for proper version control

Access control:
  - Permissions: Can Manage, Can Edit, Can Run, Can View
  - Set per notebook or per folder
  - Workspace admin can set default permissions

Modular Notebook Design for Production

Production notebooks should be modular — each notebook does one thing well, and they chain together via orchestration.

Analogy — A restaurant kitchen. The prep cook (extract notebook) chops vegetables. The line cook (transform notebook) cooks each dish. The expeditor (load notebook) plates and delivers. The head chef (orchestrator notebook) coordinates everyone. Each station works independently, follows its own recipe, and communicates through the pass (arguments and return values).

Project structure:
  /projects/orders_pipeline/
    00_config.py           -- Shared configuration and constants
    01_extract.py          -- Extract from source systems
    02_validate.py         -- Data quality checks
    03_transform.py        -- Business logic transforms
    04_load.py             -- Write to target tables
    orchestrator.py        -- Chain all steps together
    utils/
      common_functions.py  -- Reusable functions (%run this)
      quality_checks.py    -- Validation utilities

# 00_config.py -- Shared configuration
# (This notebook is %run from other notebooks)

# Environment detection
env = spark.conf.get("spark.databricks.clusterUsageTags.clusterName", "dev")
if "prod" in env.lower():
    CATALOG = "prod_catalog"
    LOG_LEVEL = "WARNING"
else:
    CATALOG = "dev_catalog"
    LOG_LEVEL = "INFO"

# Table names
BRONZE_ORDERS = f"{CATALOG}.bronze.orders"
SILVER_ORDERS = f"{CATALOG}.silver.orders_clean"
GOLD_REVENUE = f"{CATALOG}.gold.revenue_by_region"

# Processing parameters
BATCH_SIZE = 100000
QUALITY_THRESHOLD = 0.95

print(f"Config loaded: env={env}, catalog={CATALOG}")

# 01_extract.py -- Extract step
%run ./00_config
%run ./utils/common_functions

# Get parameters from widgets (set by orchestrator or manual run)
dbutils.widgets.text("processing_date", "2026-07-19")
date = dbutils.widgets.get("processing_date")

# Extract
df = spark.read.table(BRONZE_ORDERS).filter(f"order_date = '{date}'")
row_count = df.count()
print(f"Extracted {row_count} rows for {date}")

# Store in a temp view for downstream notebooks
df.createOrReplaceTempView("extracted_orders")

# Exit with a return value for the orchestrator
dbutils.notebook.exit(f'{{"rows": {row_count}, "date": "{date}"}}')

# orchestrator.py -- Chain everything together
import json
from datetime import datetime

dbutils.widgets.text("processing_date", datetime.now().strftime("%Y-%m-%d"))
date = dbutils.widgets.get("processing_date")
print(f"Pipeline start: {date}")

steps = [
    ("01_extract", {"processing_date": date}),
    ("02_validate", {"processing_date": date}),
    ("03_transform", {"processing_date": date}),
    ("04_load", {"processing_date": date}),
]

for step_name, args in steps:
    print(f"\n{'='*40}")
    print(f"Running: {step_name}")
    try:
        result = dbutils.notebook.run(f"./{step_name}", 1800, args)
        result_data = json.loads(result)
        print(f"  Result: {result_data}")
    except Exception as e:
        print(f"  FAILED: {e}")
        dbutils.notebook.exit(json.dumps({
            "status": "failed",
            "failed_step": step_name,
            "error": str(e)
        }))

print(f"\nPipeline complete: all steps succeeded")
dbutils.notebook.exit(json.dumps({"status": "success", "date": date}))

Notebook Best Practices for Data Engineers

Structuring a Production Notebook

# Cell 1: Documentation (always first)
# %md
# # Orders Pipeline -- Silver Layer
# **Owner:** Naveen Vuppula
# **Schedule:** Daily at 6 AM
# **Source:** bronze.vendor_orders
# **Target:** silver.orders_clean
# **Last updated:** 2026-07-19

# Cell 2: Configuration
%run ./00_config

# Cell 3: Parameters
dbutils.widgets.text("processing_date", "2026-07-19")
dbutils.widgets.dropdown("mode", "incremental", ["incremental", "full"])

# Cell 4: Import and setup
from pyspark.sql.functions import col, trim, lower, current_timestamp, lit

date = dbutils.widgets.get("processing_date")
mode = dbutils.widgets.get("mode")
print(f"Running: date={date}, mode={mode}")

# Cell 5: Extract
# (extraction code)

# Cell 6: Transform
# (transformation code)

# Cell 7: Validate
# (quality checks)

# Cell 8: Load
# (write to target)

# Cell 9: Summary and exit
dbutils.notebook.exit(json.dumps({"status": "success", "rows": row_count}))

Common Mistakes

1. Using %run when you need dbutils.notebook.run. %run shares variables and cannot be parallelized or error-handled independently. Use %run for shared utilities and config loading. Use dbutils.notebook.run for pipeline steps that should run in isolation with timeout and error handling.

2. Not restarting Python after %pip install. Packages installed with %pip are not available until the Python interpreter restarts. You must click “Restart Python” or call dbutils.library.restartPython() after every %pip install. Forgetting this leads to confusing “ModuleNotFoundError” errors.

3. Hardcoding values instead of using widgets. A notebook with date = "2026-07-19" hardcoded requires a code change every day. Use dbutils.widgets.text("date", ...) so the notebook can be parameterized from Workflows, the orchestrator, or manual runs without modifying code.

4. Putting all pipeline logic in one giant notebook. A 500-line notebook with extract, transform, validate, load, and alerting is impossible to test, debug, or reuse. Split into modular notebooks (one per step) and chain them with an orchestrator.

5. Using show() instead of display() for data exploration. show() prints truncated plain text. display() renders interactive tables with sorting, filtering, and built-in chart visualizations. Use show() only for quick debugging of small outputs; use display() for everything else.

6. Not documenting notebooks with %md cells. A notebook without documentation is useless to anyone else (and to you in 3 months). Add a %md cell at the top with purpose, owner, schedule, source/target tables, and parameters. Add %md cells before major sections explaining the business logic.

7. Installing libraries at the cluster level for one-off tasks. Cluster libraries affect all notebooks on that cluster. If you need a package for one notebook, use %pip install (notebook-scoped) instead. This avoids version conflicts and keeps the cluster clean.

8. Forgetting that %sh runs only on the driver node. %sh commands execute on the driver, not on worker nodes. If you need to run a command on all nodes (like installing a system library), use an init script in the cluster configuration.

Interview Questions

Q: What are magic commands in Databricks notebooks and why are they useful? A: Magic commands are prefixes (%sql, %python, %scala, %r, %md, %sh, %fs, %pip, %run) that switch the language or mode of a cell. They are useful because a single notebook can combine Python for data processing, SQL for querying, markdown for documentation, and shell commands for system operations — without switching between separate files or tools. For data engineers, the most common combination is Python cells for PySpark transformations with %sql cells for ad-hoc queries and %md cells for documentation.

Q: What is the difference between %run and dbutils.notebook.run? A: %run executes a notebook inline in the same context — all variables, functions, and imports become available in the calling notebook. It is like pasting the code directly. dbutils.notebook.run executes a notebook as a separate job with its own isolated context. It supports timeouts, parameterization via arguments, return values via dbutils.notebook.exit(), and error handling via try/except. Use %run for shared libraries and configuration. Use dbutils.notebook.run for pipeline orchestration where steps should be independent and recoverable.

Q: What are Databricks widgets and how do you use them in pipelines? A: Widgets are interactive input controls (text, dropdown, combobox, multiselect) that appear at the top of a notebook. They let you parameterize notebooks without changing code. Create with dbutils.widgets.text("name", "default", "label"), read with dbutils.widgets.get("name"). In pipelines, Databricks Workflows pass widget values as job parameters. When using dbutils.notebook.run, pass values via the arguments parameter. In SQL cells, reference widgets with :param_name syntax.

Q: What is the difference between display() and show() in Databricks? A: show() is a PySpark method that prints a truncated plain-text representation of a DataFrame (first 20 rows by default). display() is a Databricks-specific function that renders an interactive HTML table with sorting, filtering, column resizing, and built-in chart visualizations (bar, line, pie, scatter). display() also works with pandas DataFrames, images, and matplotlib figures. For data exploration and sharing results, always use display(). Use show() only for quick debugging.

Q: How do notebook-scoped libraries work and why should you use them? A: Notebook-scoped libraries are installed with %pip install package_name and are available only in the current notebook session. They do not affect other notebooks on the same cluster. This prevents version conflicts — one notebook can use pandas 2.0 while another uses 1.5 on the same cluster. After installing, you must restart the Python interpreter (dbutils.library.restartPython()). Use notebook-scoped libraries for project-specific dependencies. Use cluster libraries only for packages every notebook needs (like the Databricks runtime defaults).

Q: How would you structure a production data pipeline using Databricks notebooks? A: Create separate notebooks for each pipeline step: config (shared constants), extract (pull from source), validate (quality checks), transform (business logic), and load (write to target). Create a shared utilities notebook with reusable functions (loaded via %run). Create an orchestrator notebook that chains steps using dbutils.notebook.run with timeouts, error handling, and parameterization. Store all notebooks in a Git-connected Databricks Repo. Schedule the orchestrator via Databricks Workflows. This modular approach makes each step independently testable, debuggable, and reusable.

Q: How do you pass parameters between notebooks in a Databricks pipeline? A: Three methods. First, widgets: the calling notebook sets argument values in dbutils.notebook.run(arguments={“key”: “value”}), and the called notebook reads them with dbutils.widgets.get(“key”). Second, return values: the called notebook uses dbutils.notebook.exit(“result string”) to pass a value back to the caller. Third, shared state: notebooks can read and write to Delta tables or temporary views that act as shared storage between steps. For production pipelines, use widgets for input parameters and dbutils.notebook.exit for status reporting.

Wrapping Up

Databricks notebooks are far more than code editors — they are complete development, collaboration, and orchestration environments. Master magic commands to work in multiple languages. Use widgets to parameterize your notebooks. Build modular pipelines with %run for shared code and dbutils.notebook.run for orchestrated steps. Document everything with %md cells. And use display() and displayHTML() to make your results readable and shareable.

Related posts: — Databricks Intro & dbutils — Workflows & Jobs — Git Integration & CI/CD — File Storage (Volumes, DBFS) — Secret Scopes & Key Vault