Table of Contents
- Azure Diagnostic Settings — Streaming Logs to Log Analytics
- Azure Monitor Alerts
- System Tables — Built-in Usage Analytics
- Spark UI Deep Dive for Troubleshooting
- AI/BI Genie — Natural Language Data Discovery
- Common Mistakes
- Interview Questions
- Wrapping Up
Our Workflows & Jobs post covered scheduling and running pipelines. Our Delta Lake & PySpark Optimization post covered performance tuning. This post covers what happens after your pipelines are running: monitoring them, streaming diagnostic logs to Azure Monitor, setting up alerts, using system tables for usage analytics, reading the Spark UI for troubleshooting, and configuring AI/BI Genie for natural-language data discovery.
Analogy — A hospital patient monitoring system. Your Databricks workspace is the patient. Diagnostic logs are the vital signs (heart rate, blood pressure, oxygen levels) — continuously measured and streamed to a central monitor. Azure Monitor is the monitoring station where nurses (administrators) watch dashboards and receive alerts when values go out of range. System tables are the patient’s medical history — a long-term record of every measurement. The Spark UI is the X-ray machine — it lets you look inside a specific job to diagnose exactly what is wrong. AI/BI Genie is the patient portal — it lets non-medical staff (business users) ask questions about the data in plain language without knowing how to read an X-ray.
Azure Diagnostic Settings — Streaming Logs to Log Analytics
Azure Databricks generates diagnostic logs for every significant event: cluster starts/stops, job runs, notebook executions, SQL queries, authentication events, and security changes. By default, these logs stay inside Databricks. To query, alert on, and retain them long-term, you stream them to an Azure Log Analytics workspace using Azure Monitor diagnostic settings.
Setting Up Diagnostic Settings
Step-by-step (Azure Portal):
1. Navigate to your Databricks workspace resource in the Azure portal
2. In the left sidebar, under "Monitoring," click "Diagnostic settings"
3. Click "+ Add diagnostic setting"
4. Configure:
- Name: "databricks-to-log-analytics"
- Log categories to enable:
* accounts -- Authentication and account events
* clusters -- Cluster create, terminate, resize events
* jobs -- Job run start, end, failure events
* notebook -- Notebook command execution
* sqlPermissions -- SQL permission changes
* dbfs -- DBFS file operations
* ssh -- SSH access to clusters
* workspace -- Workspace configuration changes
* secrets -- Secret scope access
* sqlAnalytics -- SQL warehouse query events
* unityCatalog -- Unity Catalog operations
* deltaPipelines -- Lakeflow pipeline events
- Destination:
* Check "Send to Log Analytics workspace"
* Select your Log Analytics workspace
5. Click "Save"
After 5-10 minutes, logs begin flowing to Log Analytics.Log Categories Reference
| Log Category | What It Captures | DP-750 Relevance |
|---|---|---|
| **accounts** | Login, logout, token creation, permission changes | Security auditing |
| **clusters** | Create, start, terminate, resize, edit events | Cost monitoring, troubleshooting |
| **jobs** | Job run start, end, success, failure, duration | Pipeline monitoring |
| **notebook** | Command execution, cell results | Debugging, audit trail |
| **dbfs** | File create, read, delete in DBFS/Volumes | Data access auditing |
| **sqlAnalytics** | SQL warehouse queries, query duration, bytes scanned | Query performance |
| **unityCatalog** | Table create, grant, revoke, lineage events | Governance auditing |
| **deltaPipelines** | Lakeflow pipeline updates, expectations, failures | Pipeline data quality |
| **secrets** | Secret read, list, ACL changes | Security monitoring |
Querying Logs in Log Analytics (KQL)
Once logs flow to Log Analytics, you query them using KQL (Kusto Query Language). The Databricks logs land in the DatabricksClusters, DatabricksJobs, DatabricksNotebook, and other tables.
-- Find all failed job runs in the last 24 hours
DatabricksJobs
| where TimeGenerated > ago(24h)
| where ActionName == "runFailed"
| project TimeGenerated, Identity, RequestParams
| order by TimeGenerated desc
-- Cluster usage: which clusters ran the longest?
DatabricksClusters
| where TimeGenerated > ago(7d)
| where ActionName == "create" or ActionName == "delete"
| summarize TotalEvents = count() by ClusterId = tostring(RequestParams.cluster_id)
| order by TotalEvents desc
-- Who accessed what tables in Unity Catalog?
DatabricksUnityCatalog
| where TimeGenerated > ago(24h)
| where ActionName contains "getTable" or ActionName contains "selectData"
| project TimeGenerated, Identity, ActionName, RequestParams
| order by TimeGenerated desc
-- SQL warehouse query performance
DatabricksSQLAnalytics
| where TimeGenerated > ago(24h)
| where ActionName == "queryEnd"
| extend Duration = todouble(Response.duration_ms)
| summarize AvgDuration = avg(Duration), MaxDuration = max(Duration),
QueryCount = count() by bin(TimeGenerated, 1h)
| order by TimeGenerated descAzure Monitor Alerts
Azure Monitor alerts notify you when something goes wrong — a job fails, a cluster runs too long, or query latency exceeds a threshold. You define alert rules based on Log Analytics queries.
Creating an alert rule (Azure Portal):
1. In Log Analytics, run your query (e.g., failed jobs)
2. Click "New alert rule"
3. Configure:
- Condition: "Number of results greater than 0"
- Evaluation: Every 5 minutes, looking back 5 minutes
- Severity: Sev 2 (Warning) or Sev 1 (Error)
4. Action group:
- Email: data-engineering-team@company.com
- Webhook: Slack or Teams webhook URL
- Azure Function: trigger automated remediation
5. Name: "Databricks Job Failure Alert"
6. Click "Create"
Common alert scenarios:
- Job failure: DatabricksJobs | where ActionName == "runFailed"
- Cluster running > 4 hours: custom metric on cluster uptime
- SQL query > 60 seconds: DatabricksSQLAnalytics | where Duration > 60000
- Unauthorized access attempt: DatabricksAccounts | where ActionName == "loginFailed"
- Unity Catalog permission change: DatabricksUnityCatalog | where ActionName contains "grant"System Tables — Built-in Usage Analytics
Databricks system tables provide pre-built analytics on usage, billing, compute, and data quality — no Azure Monitor setup required. These live in the system catalog in Unity Catalog.
-- Billing: cost per workspace, per user
SELECT
workspace_id,
usage_date,
sku_name,
SUM(usage_quantity) AS total_dbus,
ROUND(SUM(usage_quantity) * 0.55, 2) AS estimated_cost_usd
FROM system.billing.usage
WHERE usage_date >= DATE_SUB(CURRENT_DATE(), 30)
GROUP BY workspace_id, usage_date, sku_name
ORDER BY estimated_cost_usd DESC;
-- Compute: cluster utilization
SELECT
cluster_id,
cluster_name,
driver_node_type,
worker_count,
state,
state_start_time,
state_end_time,
TIMESTAMPDIFF(MINUTE, state_start_time, state_end_time) AS duration_minutes
FROM system.compute.clusters
WHERE state = 'RUNNING'
AND state_start_time >= DATE_SUB(CURRENT_DATE(), 7)
ORDER BY duration_minutes DESC;
-- Lakeflow pipeline expectations (data quality metrics)
SELECT
pipeline_name,
expectation_name,
passed_records,
failed_records,
ROUND(failed_records * 100.0 / NULLIF(passed_records + failed_records, 0), 2) AS failure_pct
FROM system.lakeflow.events
WHERE event_type = 'expectation'
AND timestamp >= DATE_SUB(CURRENT_DATE(), 7)
ORDER BY failure_pct DESC;
-- Query history: slowest queries
SELECT
statement_id,
executed_by,
statement_text,
total_duration_ms,
rows_produced,
bytes_scanned
FROM system.query.history
WHERE start_time >= DATE_SUB(CURRENT_DATE(), 1)
ORDER BY total_duration_ms DESC
LIMIT 20;Spark UI Deep Dive for Troubleshooting
The Spark UI is your X-ray machine for diagnosing slow or failed Spark jobs. Every cluster and job has a Spark UI that shows the execution plan (DAG), stage timelines, task distribution, shuffle metrics, and storage/memory usage.
Reading the Spark UI
Accessing the Spark UI:
- Running cluster: Compute -> click cluster -> Spark UI tab
- Completed job: Workflows -> Job run -> click "Spark UI" link
Key tabs:
1. Jobs tab: Shows all Spark jobs triggered by your code
- Click a job to see its stages
- Look for: failed stages (red), slow stages (long bar)
2. Stages tab: Shows individual stages within a job
- Look for: shuffle read/write size (indicates data movement)
- Look for: task distribution (one task 10x longer = data skew)
- Look for: "Spill (Disk)" > 0 (not enough memory)
3. SQL/DataFrame tab: Shows the logical and physical plan
- Click a query to see the DAG (directed acyclic graph)
- Look for: BroadcastHashJoin (good) vs SortMergeJoin (expensive)
- Look for: Exchange nodes (shuffle operations)
4. Environment tab: Shows all Spark configuration
- Verify: spark.sql.shuffle.partitions, spark.sql.adaptive.enabled
5. Storage tab: Shows cached DataFrames/tables
- Check: cache utilization, evictionCommon Spark UI Diagnosis Patterns
| Symptom | Where to Look | Likely Cause | Fix |
|---|---|---|---|
| One task takes 10x longer than others | Stages > Task Duration | Data skew | Repartition, salting, or enable AQE skew join |
| “Spill (Disk)” in stage details | Stages > Shuffle Spill | Executor memory too small | Increase executor memory or reduce partition count |
| Massive shuffle read/write | Stages > Shuffle Read/Write | Wide transformation (join, groupBy) | Broadcast small tables, reduce shuffle partitions |
| Job takes long but CPU is idle | Jobs > Timeline | Too few partitions | Increase partitions or use AQE |
| OOM (Out of Memory) error | Driver/Executor logs | Large collect(), broadcast, or skew | Avoid collect() on large data, increase memory |
AI/BI Genie — Natural Language Data Discovery
AI/BI Genie is Databricks’ natural-language interface that lets business users ask questions about data in plain English — no SQL required. Data engineers configure Genie instructions that tell Genie which tables to use, how to interpret business terms, and what joins are appropriate.
Analogy — Training a new intern. Genie is an intelligent intern who knows SQL perfectly but does not know your business. Instructions are the onboarding guide: “when someone asks about revenue, use the gold_revenue table,” “region codes map to this dimension table,” “fiscal year starts in April.” Without instructions, the intern writes technically correct but wrong SQL. With instructions, they answer business questions accurately.
Setting Up Genie
Step-by-step:
1. Navigate to a SQL Warehouse dashboard or Genie space
2. Create a new Genie space:
- Name: "Sales Analytics Genie"
- Tables: Select the Unity Catalog tables Genie can access
* gold_catalog.analytics.revenue_by_region
* gold_catalog.analytics.customer_segments
* gold_catalog.dimensions.date_dim
3. Add instructions (natural language):
- "Revenue means the 'total_revenue' column in the revenue_by_region table"
- "When asked about YTD, filter by order_date >= first day of current year"
- "Region names should be displayed as full names, not codes"
- "Always exclude test orders where is_test = true"
- "Fiscal year starts April 1st"
4. Add sample questions:
- "What was total revenue by region last quarter?"
- "Show me the top 10 customers by lifetime value"
- "How does this month compare to the same month last year?"
5. Publish the Genie space
Users interact:
User: "What was revenue in Ontario last month?"
Genie: Generates SQL, runs it, returns formatted result
User: "Break that down by product category"
Genie: Modifies query, adds GROUP BY product_categoryGenie Instructions Best Practices
DO:
- Define business terms: "revenue = SUM(amount) from the orders table"
- Specify joins: "to get customer name, join orders to customers on customer_id"
- Set filters: "always exclude where status = 'cancelled'"
- Define date logic: "fiscal year starts April 1"
- Provide sample questions with expected SQL
DON'T:
- Give Genie access to raw/bronze tables (messy, no business meaning)
- Skip defining business terms (Genie will guess wrong)
- Expose PII tables without column masks
- Trust Genie for critical reporting without validationCommon Mistakes
1. Not enabling diagnostic settings. Without diagnostic settings, Databricks logs exist only inside the platform with limited retention. Streaming to Log Analytics gives you long-term retention, cross-resource correlation, and alerting. Enable on every production workspace.
2. Enabling all log categories without filtering. Some categories (like notebook and dbfs) generate high volumes. Start with jobs, clusters, unityCatalog, and sqlAnalytics — the most actionable categories. Add others as needed.
3. Not setting up alerts for job failures. A failed job at 2 AM with no alert means stale dashboards all morning. At minimum, alert on DatabricksJobs | where ActionName == "runFailed" with email/Slack notification.
4. Ignoring system tables. System tables (system.billing.usage, system.compute.clusters, system.query.history) provide free, built-in analytics without Azure Monitor setup. Query them regularly for cost optimization and performance monitoring.
5. Not reading the Spark UI for slow jobs. Adding more nodes to a skewed job does not fix the skew. The Spark UI shows exactly which stage is slow and why (skew, spill, shuffle). Always check the Spark UI before scaling up.
6. Configuring Genie without instructions. Genie without instructions is like a GPS without a map database — it knows how to navigate but has no idea where things are. Always provide business term definitions, join paths, and default filters.
7. Giving Genie access to bronze tables. Bronze tables have raw column names, inconsistent types, and no business meaning. Genie works best on gold-layer tables with clean schemas, meaningful column names, and documented descriptions.
8. Not testing Genie with sample questions. Before publishing a Genie space, test it with 10-20 real business questions. Verify the generated SQL is correct. Refine instructions based on where Genie gets confused.
Interview Questions
Q: How do you stream Databricks diagnostic logs to Azure Monitor? A: In the Azure portal, navigate to your Databricks workspace resource, go to Monitoring > Diagnostic settings, and add a new diagnostic setting. Select the log categories you want (jobs, clusters, unityCatalog, sqlAnalytics), choose “Send to Log Analytics workspace,” select your workspace, and save. Logs begin flowing within minutes. You can then query them with KQL in Log Analytics and set up alert rules.
Q: What are the key Databricks log categories and what does each capture? A: The most important categories are: jobs (job run start, end, failure — for pipeline monitoring), clusters (create, terminate, resize — for cost monitoring), unityCatalog (grants, table operations — for governance auditing), sqlAnalytics (SQL warehouse queries — for performance monitoring), and accounts (login, logout, token creation — for security auditing). Each category maps to a table in Log Analytics (DatabricksJobs, DatabricksClusters, etc.).
Q: What are Databricks system tables and how do they differ from Azure Monitor logs? A: System tables are built-in Unity Catalog tables (system.billing.usage, system.compute.clusters, system.query.history, system.lakeflow.events) that provide usage and performance data without external configuration. Azure Monitor logs require diagnostic settings to be configured but provide cross-resource correlation, long-term retention, and alerting capabilities. Use system tables for quick internal analytics; use Azure Monitor for enterprise monitoring, alerting, and compliance.
Q: How do you diagnose a slow Spark job using the Spark UI? A: Open the Spark UI for the job run. Check the Stages tab for the slowest stage. Look for: one task taking much longer than others (data skew), Spill (Disk) greater than zero (not enough memory), large Shuffle Read/Write (expensive data movement), or Exchange nodes in the SQL DAG (shuffle operations). For skew, repartition or enable AQE. For spill, increase executor memory. For large shuffles, broadcast small tables or reduce shuffle partitions.
Q: What is AI/BI Genie and how do you configure it for a data team? A: AI/BI Genie is a natural-language interface that lets business users query data by asking questions in plain English. You configure it by creating a Genie space, selecting the Unity Catalog tables it can access (preferably gold-layer tables), and adding instructions that define business terms, join paths, default filters, and date logic. Instructions are critical — without them, Genie generates technically valid but semantically wrong SQL.
Q: How would you set up an alert for Databricks job failures? A: In Log Analytics, write a KQL query: DatabricksJobs | where ActionName == "runFailed" | where TimeGenerated > ago(5m). Click “New alert rule.” Set the condition to “Number of results greater than 0,” evaluation every 5 minutes. Create an action group with email notification to the data engineering team and optionally a webhook to Slack/Teams. Name the alert “Databricks Job Failure” and save. The team receives an alert within minutes of any job failure.
Q: What is the difference between Databricks-native monitoring and Azure Monitor? A: Databricks-native monitoring includes the Spark UI (job-level debugging), Monitoring Hub (real-time pipeline status), and system tables (usage analytics). Azure Monitor provides enterprise-grade capabilities: long-term log retention (90+ days), cross-resource correlation (correlate Databricks logs with ADF, Azure SQL, etc.), KQL-powered alerting, and integration with Azure dashboards and ITSM tools. Production workspaces should use both: Databricks-native for day-to-day operations, Azure Monitor for enterprise monitoring and compliance.
Wrapping Up
Monitoring is not optional in production data engineering. Enable diagnostic settings on every workspace to stream logs to Log Analytics. Set up alerts for job failures and slow queries. Query system tables weekly for cost and performance trends. Use the Spark UI to diagnose slow jobs before throwing more compute at them. And configure AI/BI Genie on your gold-layer tables to give business users self-service data access without burdening the engineering team with ad-hoc SQL requests.
Related posts: — Workflows & Jobs — Delta Lake & PySpark Optimization — Unity Catalog Deep Dive — Databricks SQL & Warehouses — DP-750 Study Guide
Naveen Vuppula is a Senior Data Engineering Consultant and app developer based in Ontario, Canada. He writes about Python, SQL, AWS, Azure, and everything data engineering at DriveDataScience.com.