Managed vs External in Databricks: The Complete Guide to Tables, Volumes, Storage Locations, External Locations, Storage Credentials, the MANAGE Privilege, and Every Context Where These Words Appear

Table of Contents

• Why This Post Exists — The Confusion
• What “Managed” Actually Means in Databricks
• Managed vs External Tables
• Managed vs External Volumes
• Managed Storage Locations — Where “Managed” Data Actually Lives
• External Locations — Registered Paths to Your Storage
• Storage Credentials — The Keys Behind Everything
• The MANAGE Privilege — A Third Meaning
• Hive Metastore Managed vs Unity Catalog Managed — Different Behaviors
• The Complete Decision Framework
• How All the Pieces Connect — The Full Picture
• Temporary Views and Global Temporary Views
• Common Mistakes
• Interview Questions
• Wrapping Up

In Databricks, the words managed and external appear in at least six different contexts — managed tables, external tables, managed volumes, external volumes, managed storage locations, external locations, storage credentials, and even the MANAGE privilege. Each has a slightly different meaning, and confusing them is one of the most common mistakes in Databricks interviews and production setups.

This post maps every use of “managed” and “external” in Databricks into one clear picture, explains how they connect, and gives you a decision framework for choosing between them.

Analogy — A parking garage. Imagine you own a car (your data). A managed table is like valet parking — you hand over the keys (data), the garage (Unity Catalog) decides which spot to use, and when you say “destroy my account” (DROP TABLE), they crush the car too. An external table is like self-parking — you choose the spot yourself (LOCATION), and when you cancel your parking permit (DROP TABLE), the garage removes your name from the system, but your car stays exactly where you parked it. Both use the same garage building (cloud storage). Both require a key card (storage credential). The difference is who controls the parking spot and what happens when you leave.

Why This Post Exists — The Confusion

Places where "managed" and "external" appear in Databricks:

  1. Managed TABLE vs External TABLE
     "Who controls where the data files live?"

  2. Managed VOLUME vs External VOLUME
     "Who controls where the non-tabular files live?"

  3. Managed STORAGE LOCATION
     "The default path where managed objects are stored"

  4. EXTERNAL LOCATION
     "A registered cloud storage path for external objects"

  5. STORAGE CREDENTIAL
     "The IAM role / service principal that grants access to cloud storage"

  6. MANAGE privilege
     "A Unity Catalog permission -- completely unrelated to managed/external objects"

  7. Hive Metastore "managed" vs Unity Catalog "managed"
     "Different DROP behaviors, different storage ownership"

  The word "managed" means something different in EACH context.
  This post untangles all of them.

What “Managed” Actually Means in Databricks

The official Databricks documentation says it clearly: “managed” refers to lifecycle management, not physical location. In both managed and external objects, your data lives in YOUR cloud storage account (S3, ADLS Gen2, or GCS). Databricks never takes ownership of your data. The difference is:

• Managed: Unity Catalog decides WHERE in your storage the data goes, HOW it is organized, and DELETES it when you DROP the object
• External: YOU decide where the data goes (you provide a LOCATION), and when you DROP the object, only the metadata is removed — the data files remain

Common misconception:
  "Managed means Databricks stores the data"
  WRONG. Your data ALWAYS lives in YOUR cloud storage account.

  "Managed means I lose control of my data"
  WRONG. Managed means Unity Catalog handles the directory structure
  and file lifecycle. You still own the underlying storage.

  "External means the data is outside Databricks"
  PARTIALLY CORRECT. External means YOU control the storage location.
  The data might be right next to managed data in the same storage account.

Managed vs External Tables

Tables are the most common context for managed vs external. Both store structured data as Delta Lake files. The difference is lifecycle control.

Managed Tables

-- Managed table: no LOCATION clause
-- Unity Catalog chooses the storage path automatically
CREATE TABLE prod_catalog.silver.customers (
  customer_id BIGINT,
  customer_name STRING,
  email STRING,
  region STRING
);

-- Where does the data actually go?
DESCRIBE DETAIL prod_catalog.silver.customers;
-- location: abfss://unity-catalog@storageaccount.dfs.core.windows.net/
--           __unitystorage/schemas/<UUID>/tables/<UUID>/

-- Key behavior: DROP = data DELETED
DROP TABLE prod_catalog.silver.customers;
-- Metadata: REMOVED from Unity Catalog
-- Data files: DELETED (after retention period, recoverable via UNDROP)

External Tables

-- External table: YOU specify the LOCATION
CREATE TABLE prod_catalog.bronze.raw_orders (
  order_id BIGINT,
  amount DECIMAL(10,2),
  order_date DATE
)
LOCATION 'abfss://raw-data@storageaccount.dfs.core.windows.net/orders/';

-- Key behavior: DROP = metadata removed, data STAYS
DROP TABLE prod_catalog.bronze.raw_orders;
-- Metadata: REMOVED from Unity Catalog
-- Data files: UNTOUCHED (still in your storage, accessible by other tools)

Side-by-Side Comparison

Analogy — Renting vs owning an apartment. A managed table is like renting — the landlord (Unity Catalog) picks the unit, maintains it, and if you end the lease (DROP), the apartment is emptied. An external table is like owning — you picked the apartment yourself, and if you cancel your property management service (DROP), the apartment and everything in it stays yours.

Managed vs External Volumes

Volumes are Unity Catalog’s governed storage for non-tabular files (CSV, JSON, images, PDFs, ML artifacts). The managed vs external distinction works the same way as tables.

Managed Volumes

-- Managed volume: no LOCATION clause
CREATE VOLUME prod_catalog.bronze.raw_files
  COMMENT 'Raw CSV and JSON files from vendors';

-- Access files
-- /Volumes/prod_catalog/bronze/raw_files/orders.csv

-- DROP = files deleted
DROP VOLUME prod_catalog.bronze.raw_files;
-- Files: DELETED from managed storage

External Volumes

-- External volume: YOU specify the LOCATION
CREATE EXTERNAL VOLUME prod_catalog.bronze.vendor_uploads
  LOCATION 'abfss://raw-data@storageaccount.dfs.core.windows.net/vendor-uploads/'
  COMMENT 'Vendor file upload location';

-- Access files (same path pattern)
-- /Volumes/prod_catalog/bronze/vendor_uploads/orders.csv

-- DROP = metadata removed, files stay
DROP VOLUME prod_catalog.bronze.vendor_uploads;
-- Files: UNTOUCHED in your storage

Volumes Comparison

The Confusing Part — Do Managed Volumes Need External Locations?

This is the question that confuses everyone. The answer depends on the context:

Managed volumes:
  - Do NOT require you to create an external location
  - But they DO require a managed storage location to be configured
    at the metastore, catalog, or schema level
  - Under the hood, Unity Catalog uses a storage credential to access
    the managed storage location
  - You configure this ONCE, then all managed objects use it automatically

External volumes:
  - Require a storage credential (IAM role / service principal)
  - Require an external location (registered path + credential)
  - You specify the LOCATION pointing to a path within that external location

So both ultimately need credentials to access cloud storage.
The difference: managed sets it up ONCE at the metastore/catalog level.
External requires per-object LOCATION specification.

Managed Storage Locations — Where “Managed” Data Actually Lives

A managed storage location is the default cloud storage path where Unity Catalog stores all managed objects (managed tables and managed volumes). It can be configured at three levels, with the most specific level taking priority.

Managed storage location hierarchy (most specific wins):

  Schema level:   CREATE SCHEMA silver MANAGED LOCATION 'abfss://...'
    ↑ wins if set
  Catalog level:  CREATE CATALOG prod MANAGED LOCATION 'abfss://...'
    ↑ wins if schema has no managed location
  Metastore level: configured during metastore creation
    ↑ default fallback

Example:
  Metastore managed location: abfss://unity@storage.dfs.core.windows.net/metastore/
  Catalog managed location:   abfss://unity@storage.dfs.core.windows.net/prod/
  Schema managed location:    abfss://unity@storage.dfs.core.windows.net/prod/silver/

  A managed table in prod.silver.customers would be stored under:
    abfss://unity@storage.dfs.core.windows.net/prod/silver/__unitystorage/...

  A managed table in prod.bronze.raw_orders (no schema-level location):
    abfss://unity@storage.dfs.core.windows.net/prod/__unitystorage/...

-- Set managed location at catalog level
CREATE CATALOG prod_catalog
  MANAGED LOCATION 'abfss://unity-catalog@storageaccount.dfs.core.windows.net/prod/';

-- Set managed location at schema level
CREATE SCHEMA prod_catalog.silver
  MANAGED LOCATION 'abfss://unity-catalog@storageaccount.dfs.core.windows.net/prod/silver/';

-- Check managed location
DESCRIBE CATALOG prod_catalog;
DESCRIBE SCHEMA prod_catalog.silver;

External Locations — Registered Paths to Your Storage

An external location is a Unity Catalog securable object that maps a cloud storage path to a storage credential. It tells Unity Catalog: “this specific path in cloud storage can be accessed using this specific credential.”

Analogy — A registered delivery address. The storage credential is your master key to a building. The external location is a registered address within that building — “Suite 301, 123 Data Street.” When you create an external table at that address, Unity Catalog knows which key opens that door.

-- Create an external location
CREATE EXTERNAL LOCATION prod_raw_data
  URL 'abfss://raw-data@storageaccount.dfs.core.windows.net/'
  WITH (STORAGE CREDENTIAL az_storage_cred)
  COMMENT 'Production raw data lake';

-- Now you can create external tables and volumes at paths under this location
CREATE TABLE prod_catalog.bronze.orders
  LOCATION 'abfss://raw-data@storageaccount.dfs.core.windows.net/orders/';
  -- Unity Catalog matches this path to prod_raw_data external location
  -- and uses az_storage_cred for authentication

-- List external locations
SHOW EXTERNAL LOCATIONS;

External locations are needed for:
  - Creating external tables (LOCATION clause)
  - Creating external volumes (LOCATION clause)
  - Reading files from cloud storage (READ FILES privilege)
  - Writing files to cloud storage (WRITE FILES privilege)

External locations are NOT needed for:
  - Creating managed tables (managed storage location is used instead)
  - Creating managed volumes (managed storage location is used instead)

Storage Credentials — The Keys Behind Everything

A storage credential is the authentication mechanism (Azure managed identity, AWS IAM role, or GCS service account) that Databricks uses to access cloud storage. Both managed storage locations and external locations use storage credentials.

-- Create a storage credential (Azure)
CREATE STORAGE CREDENTIAL az_storage_cred
  WITH AZURE_MANAGED_IDENTITY (
    ACCESS_CONNECTOR_ID = '/subscriptions/.../accessConnectors/unity-connector'
  );

-- Create a storage credential (AWS)
CREATE STORAGE CREDENTIAL aws_storage_cred
  WITH AWS_IAM_ROLE (
    ROLE_ARN = 'arn:aws:iam::123456789012:role/unity-catalog-role'
  );

-- List storage credentials
SHOW STORAGE CREDENTIALS;

How storage credentials connect to everything:

  Storage Credential (the master key)
    |
    ├── Used by: Managed Storage Location (metastore/catalog/schema level)
    |     └── Accessed by: Managed Tables, Managed Volumes
    |
    └── Used by: External Location (registered path + credential)
          └── Accessed by: External Tables, External Volumes

The MANAGE Privilege — A Third Meaning

Just to add to the confusion, MANAGE is also a Unity Catalog privilege. It has nothing to do with managed vs external objects. The MANAGE privilege lets a user administer an object without being its owner.

-- Grant MANAGE on a schema (allows user to manage grants, transfer ownership, delete)
GRANT MANAGE ON SCHEMA prod_catalog.silver TO `platform-admin-team`;

-- MANAGE allows:
--   Granting and revoking privileges on the object
--   Transferring ownership of the object
--   Deleting the object
--   Without being the owner

-- This is a PERMISSION concept, not a storage concept
-- A user with MANAGE on a schema can manage both managed AND external tables in it

The three meanings of "managed" in Databricks:

  1. Managed TABLE/VOLUME = Unity Catalog controls storage lifecycle
     (WHERE data lives + DELETE on DROP)

  2. Managed STORAGE LOCATION = default path for managed objects
     (configured at metastore/catalog/schema level)

  3. MANAGE PRIVILEGE = permission to administer an object
     (grant/revoke privileges, transfer ownership, delete)

  Each is independent. They do not imply each other.

Hive Metastore Managed vs Unity Catalog Managed — Different Behaviors

If your workspace still has the legacy hive_metastore catalog, be aware that “managed” means something slightly different there:

-- Hive Metastore managed table (legacy)
USE hive_metastore.default;
CREATE TABLE orders (id INT, amount DOUBLE);
-- Data stored in: dbfs:/user/hive/warehouse/orders/

-- Unity Catalog managed table (current)
USE prod_catalog.silver;
CREATE TABLE orders (id INT, amount DOUBLE);
-- Data stored in: abfss://unity@storage.dfs.core.windows.net/prod/silver/__unitystorage/.../

-- Key difference: Hive managed = DBFS (Databricks controls)
--                 UC managed = YOUR cloud storage (you own, UC manages lifecycle)

Analogy — Company laptop vs personal laptop with managed software. Hive Metastore managed is like a company laptop — the company owns the hardware (DBFS) and everything on it. Unity Catalog managed is like your personal laptop with managed software — you own the laptop (cloud storage), but the IT department (Unity Catalog) installs, updates, and removes software (data) for you. When IT uninstalls an app (DROP), the app is gone, but the laptop is still yours.

The Complete Decision Framework

Decision tree: Managed or External?

  FOR TABLES:
    Is the data produced entirely within Databricks?
      YES -> Is simplicity the priority?
               YES -> Managed Table (simplest, auto-optimized)
               NO  -> External Table (you control the path)
      NO  -> External Table (data produced by Spark/Kafka/CDC outside Databricks)

    Does another engine (Spark on EMR, Snowflake, Trino) read this data directly?
      YES -> External Table (direct storage access possible)
      NO  -> Either works (managed is simpler)

    Must data survive a DROP TABLE?
      YES -> External Table (data files remain after DROP)
      NO  -> Either works (managed supports UNDROP within retention)

  FOR VOLUMES:
    Are the files produced and consumed only within Databricks?
      YES -> Managed Volume (simplest setup)
      NO  -> External Volume (files shared with other tools)

    Do the files already exist in a specific cloud storage location?
      YES -> External Volume (register the existing location)
      NO  -> Managed Volume (Unity Catalog picks the path)

  DEFAULT CHOICE:
    New projects, Databricks-only: Managed (simpler, less config)
    Multi-engine environments: External (interoperability, data persistence)

How All the Pieces Connect — The Full Picture

The complete architecture:

  Cloud Storage Account (ADLS Gen2 / S3 / GCS)
    |
    ├── Managed Storage Location
    |     (configured at metastore/catalog/schema level)
    |     (uses a storage credential for access)
    |     |
    |     ├── Managed Tables → Unity Catalog controls path + lifecycle
    |     └── Managed Volumes → Unity Catalog controls path + lifecycle
    |
    └── External Locations
          (registered paths, each with a storage credential)
          |
          ├── External Tables → YOU control path, UC controls metadata
          └── External Volumes → YOU control path, UC controls metadata

  Storage Credential
    (IAM role / managed identity / service principal)
    |
    ├── Referenced by: Managed Storage Locations
    └── Referenced by: External Locations

  Unity Catalog Privileges
    |
    ├── MANAGE → administer any object (unrelated to managed/external)
    ├── SELECT → read data from tables/views
    ├── MODIFY → write data to tables
    ├── READ VOLUME → read files from volumes
    ├── WRITE VOLUME → write files to volumes
    └── CREATE EXTERNAL TABLE/VOLUME → create external objects at a location

Temporary Views and Global Temporary Views

Beyond managed and external tables, Databricks also has temporary views and global temporary views — objects that exist only in memory and disappear when the session or cluster ends. They are not registered in Unity Catalog and have no underlying data files.

Analogy — A whiteboard vs a printed report. A managed table is a printed report filed in a cabinet (persists forever). An external table is a report filed in YOUR cabinet (persists in your storage). A temporary view is a whiteboard drawing in YOUR meeting room (visible only to you, erased when you leave). A global temporary view is a whiteboard in the SHARED conference room (visible to everyone on the cluster, erased when the room closes).

Temporary Views (Session-Scoped)

A temporary view exists only for the duration of your SparkSession (your notebook session). Other notebooks on the same cluster cannot see it. When you detach from the cluster or the session ends, the view disappears.

-- SQL: Create a temporary view
CREATE OR REPLACE TEMP VIEW active_orders AS
  SELECT * FROM prod_catalog.silver.orders WHERE status = 'active';

-- Query it (same notebook only)
SELECT region, COUNT(*) FROM active_orders GROUP BY region;

-- Another notebook on the same cluster CANNOT see active_orders

# PySpark: Create a temporary view from a DataFrame
df = spark.read.table("prod_catalog.silver.orders")
df_active = df.filter("status = 'active'")

# Register as a temp view
df_active.createOrReplaceTempView("active_orders")

# Now you can query it with SQL in the same notebook
spark.sql("SELECT region, COUNT(*) FROM active_orders GROUP BY region").show()

# Temp view disappears when the session ends

Global Temporary Views (Cluster-Scoped)

A global temporary view is visible to ALL notebooks attached to the same cluster. It lives in a special schema called global_temp. When the cluster is terminated or restarted, all global temporary views disappear.

-- SQL: Create a global temporary view
CREATE OR REPLACE GLOBAL TEMP VIEW shared_customers AS
  SELECT customer_id, customer_name, region
  FROM prod_catalog.silver.customers
  WHERE is_active = TRUE;

-- Query it (must use global_temp schema prefix)
SELECT * FROM global_temp.shared_customers;

-- ANY notebook on the same cluster can query global_temp.shared_customers

# PySpark: Create a global temporary view
df_customers = spark.read.table("prod_catalog.silver.customers")
df_active = df_customers.filter("is_active = TRUE")

# Register as a global temp view
df_active.createOrReplaceGlobalTempView("shared_customers")

# Query from any notebook on the same cluster
spark.sql("SELECT * FROM global_temp.shared_customers").show()

# Global temp view disappears when the cluster restarts

Complete Object Type Comparison

When to use each: Use managed or external tables for data that must persist. Use standard views for reusable SQL logic. Use temporary views for intermediate transformations within a notebook (like CTEs but reusable across cells). Use global temporary views when multiple notebooks on the same cluster need to share intermediate results without writing to a table.

Common Mistakes

1. Thinking “managed” means Databricks owns your data. In Unity Catalog, managed tables and volumes store data in YOUR cloud storage account. “Managed” means Unity Catalog controls the directory structure and file lifecycle. Your data never leaves your account. This is different from the legacy Hive Metastore where managed tables lived in DBFS (Databricks-controlled storage).

2. Using external tables for everything “just to be safe.” External tables require managing external locations, storage credentials, and directory structures yourself. If your data is produced entirely within Databricks and no external engine reads it, managed tables are simpler, auto-optimized, and fully governed. The “data survives DROP” argument is weakened by UNDROP support in Unity Catalog.

3. Confusing External Locations with External Tables. An external location is a Unity Catalog securable object that registers a cloud storage path. An external table is a table whose data files live at a path you specify. External tables require an external location to exist at their path, but they are different objects with different purposes.

4. Not understanding that managed objects still need storage credentials. Managed tables and volumes do not require explicit external location setup, but they DO require a managed storage location to be configured (at metastore, catalog, or schema level), which uses a storage credential. The difference is you configure this once, not per object.

5. Confusing the MANAGE privilege with managed tables. MANAGE is a Unity Catalog permission that lets non-owners administer objects (grant/revoke privileges, transfer ownership, delete). It has nothing to do with whether a table is managed or external. You can MANAGE an external table. You can lack MANAGE on a managed table.

6. Assuming Hive Metastore managed behavior applies to Unity Catalog. In the legacy Hive Metastore, managed tables lived in DBFS (Databricks infrastructure). In Unity Catalog, managed tables live in your cloud storage at the managed storage location. The word “managed” changed meaning between the two systems. If you are migrating from HMS to UC, do not assume DROP behavior is identical.

7. Creating external tables without registering an external location first. If you specify a LOCATION that is not within a registered external location, the CREATE TABLE fails with a permissions error. Always create the storage credential and external location before creating external tables or volumes at that path.

8. Not setting managed storage locations at the schema or catalog level. If you only configure the managed storage location at the metastore level, all managed objects across all catalogs and schemas end up in the same root path. Set managed storage locations at the catalog level (at minimum) to separate environments, and at the schema level to separate data domains.

Interview Questions

Q: What does “managed” mean in Databricks, and how many different meanings does it have? A: “Managed” has three distinct meanings. First, a managed table or volume is one where Unity Catalog controls the storage path and file lifecycle — DROP deletes the data. Second, a managed storage location is the configured cloud storage path where managed objects are stored, set at the metastore, catalog, or schema level. Third, the MANAGE privilege is a Unity Catalog permission that lets non-owners administer objects. All three are independent concepts. Importantly, managed objects always live in customer-owned cloud storage — Databricks never owns the data.

Q: What happens when you DROP a managed table vs an external table? A: Dropping a managed table removes both the metadata from Unity Catalog AND the underlying data files from cloud storage (recoverable via UNDROP within the retention period). Dropping an external table removes only the metadata from Unity Catalog — the underlying data files remain untouched at the specified LOCATION. This makes external tables suitable for data that must persist independently of the table registration.

Q: Do managed tables and volumes require storage credentials and external locations? A: Managed objects require a managed storage location (configured at metastore, catalog, or schema level) and the storage credential associated with it, but you do NOT create explicit external locations for them. You configure the managed storage location once, and all managed objects use it automatically. External objects require both a storage credential and an explicitly registered external location at the path where data will be stored.

Q: How do managed tables in the Hive Metastore differ from managed tables in Unity Catalog? A: In the legacy Hive Metastore, managed tables store data in DBFS — storage controlled by Databricks infrastructure. In Unity Catalog, managed tables store data in your own cloud storage account at the configured managed storage location. Both delete data on DROP, but Unity Catalog supports UNDROP for recovery. Unity Catalog also provides governance (row/column security, lineage, audit logging) that the Hive Metastore lacks.

Q: When should you use managed tables vs external tables? A: Use managed tables when data is produced and consumed only within Databricks, simplicity is the priority, and you want automatic storage optimization. Use external tables when data is shared with other engines (Spark on EMR, Snowflake, Trino), when data is produced by systems outside Databricks, when data must survive a DROP TABLE, or when regulatory requirements dictate specific storage locations. For new Databricks-only projects, managed tables are the recommended default.

Q: What is an external location and how does it relate to external tables? A: An external location is a Unity Catalog securable object that combines a cloud storage path with a storage credential, authorizing access to that path. When you create an external table with a LOCATION clause, Unity Catalog matches the path to a registered external location and uses its credential for authentication. External tables cannot be created at paths that are not within a registered external location. This ensures all cloud storage access is governed and auditable.

Q: What is the MANAGE privilege and how does it differ from managed tables? A: The MANAGE privilege is a Unity Catalog permission that allows a user to grant/revoke privileges, transfer ownership, and delete an object without being its owner. It has no relationship to managed vs external tables. A user can have MANAGE on an external table or lack MANAGE on a managed table. The word “manage” in the privilege refers to administrative control over the object, not storage lifecycle control.

Wrapping Up

The word “managed” in Databricks carries three distinct meanings: lifecycle control (managed tables and volumes), storage configuration (managed storage locations), and administrative permissions (MANAGE privilege). The word “external” carries two: user-controlled storage (external tables and volumes) and registered cloud paths (external locations). Understanding which meaning applies in which context eliminates the most common confusion in Databricks architecture discussions.

The key insight: both managed and external objects store data in YOUR cloud storage. The difference is who controls the directory structure (Unity Catalog vs you) and what happens when you DROP the object (data deleted vs data preserved). For new Databricks-only projects, managed is simpler. For multi-engine environments, external is more flexible. Most production deployments use both.

Related posts: – Unity Catalog Complete Reference – Unity Catalog Deep Dive – File Storage (Volumes, DBFS) – Managed vs External Tables – Snowflake Account Setup & RBAC

Naveen Vuppula is a Senior Data Engineering Consultant and app developer based in Ontario, Canada. He writes about Python, SQL, AWS, Azure, and everything data engineering at DriveDataScience.com.