Real-Time Intelligence in Microsoft Fabric: Eventstream, Eventhouse, KQL Database, Real-Time Dashboards, and Processing Millions of Events Per Second

Everything we have built so far — pipelines, notebooks, dataflows — operates on batch data. Run a pipeline, process a chunk, wait, repeat. But what about data that never stops flowing? IoT sensors sending readings every second. Website clickstreams generating millions of events per hour. Financial transactions that need fraud detection in milliseconds, not minutes.

That is where Real-Time Intelligence comes in — Fabric’s streaming workload that ingests, stores, queries, and visualizes continuous data in near real-time.

Think of batch processing like mail delivery — letters are collected, sorted at the post office, and delivered once a day. Real-time intelligence is like a phone call — the voice travels instantly, continuously, with no batching or waiting. Both deliver information, but phone calls (streaming) are for urgent, time-sensitive communication.

Table of Contents

• What Is Real-Time Intelligence in Fabric?
• The Four Components
• Eventstream: Ingesting Streaming Data
• What Is an Eventstream
• Supported Sources
• Creating an Eventstream
• Adding Sources
• Adding Destinations
• Transformations in Eventstream
• Eventhouse: Storing Event Data
• What Is an Eventhouse
• Creating an Eventhouse
• KQL Database Inside an Eventhouse
• KQL Database: Querying Events
• What Is KQL (Kusto Query Language)
• KQL vs SQL Comparison
• Essential KQL Queries
• Filtering and Projecting
• Aggregation and Time-Based Analysis
• Time Series Analysis
• Anomaly Detection
• Real-Time Dashboards
• Creating a Real-Time Dashboard
• Tiles and Visualizations
• Auto-Refresh
• Data Activator (Alerts and Actions)
• Setting Up Alerts
• Trigger Actions on Data Changes
• Real-World Scenario 1: IoT Factory Monitoring
• Real-World Scenario 2: E-Commerce Clickstream Analytics
• Real-World Scenario 3: Financial Transaction Monitoring
• Real-World Scenario 4: Application Log Analytics
• Batch + Real-Time: The Complete Architecture
• Real-Time Intelligence vs Azure Stream Analytics vs Event Hubs
• Common Mistakes
• Interview Questions
• Wrapping Up

What Is Real-Time Intelligence in Fabric?

Real-Time Intelligence is Fabric’s workload for continuous data processing — ingest streaming events, store them in an optimized time-series database, query with KQL (Kusto Query Language), and visualize on auto-refreshing dashboards.

STREAMING DATA                  FABRIC REAL-TIME INTELLIGENCE
┌──────────────┐
│ IoT Sensors  │──►┌────────────┐    ┌───────────┐    ┌──────────┐    ┌───────────┐
│ Clickstream  │──►│ EVENTSTREAM │───►│ EVENTHOUSE │───►│ KQL      │───►│ REAL-TIME │
│ App Logs     │──►│ (ingest)   │    │ (store)    │    │ DATABASE │    │ DASHBOARD │
│ Transactions │──►│            │    │            │    │ (query)  │    │ (visualize)│
└──────────────┘   └────────────┘    └───────────┘    └──────────┘    └───────────┘
      |                 |                  |                |                |
   Millions         Route &            Optimized        KQL queries      Auto-refresh
   events/sec       transform          time-series      in seconds       every 30 sec

The Four Components

Component	What It Does	Analogy
Eventstream	Ingests and routes streaming data	Water pipe system — collects from multiple sources, routes to destinations
Eventhouse	Stores event data at scale (petabytes)	A massive warehouse — organized for time-series data
KQL Database	Queries event data with Kusto Query Language	The librarian — finds exactly what you need from billions of records in seconds
Real-Time Dashboard	Visualizes live data with auto-refresh	The control room screens — always showing the latest data

Eventstream: Ingesting Streaming Data

What Is an Eventstream

An Eventstream is a no-code streaming pipeline — it connects to data sources, optionally transforms events in-flight, and routes them to one or more destinations. Think of it as ADF for streaming — same concept (source → transform → destination), but continuous instead of batch.

Supported Sources

Source	Type	Example
Azure Event Hubs	Streaming	IoT device events, clickstream, logs
Azure IoT Hub	IoT	Sensor readings from IoT devices
Custom App	SDK	Your application sends events via SDK
Sample Data	Testing	Built-in sample streams for learning
Azure SQL DB CDC	Change Data	Real-time changes from SQL tables
Azure Cosmos DB CDC	Change Data	Real-time changes from Cosmos containers
PostgreSQL CDC	Change Data	Real-time changes from PostgreSQL
MySQL CDC	Change Data	Real-time changes from MySQL
Google Cloud Pub/Sub	Cross-cloud	Events from Google Cloud
Amazon Kinesis	Cross-cloud	Events from AWS
Confluent Kafka	Messaging	Events from Kafka clusters

Supported Destinations

Destination	What Happens
Eventhouse / KQL Database	Events stored for KQL querying
Lakehouse	Events land as Delta tables (for Spark analysis)
Custom App	Events forwarded to your application
Reflex (Data Activator)	Events trigger alerts and actions
Derived Eventstream	Route to another eventstream (chaining)

Creating an Eventstream

1. + New item → Eventstream
2. Name: ES_IoT_Sensor_Events
3. The visual canvas opens (similar to ADF pipeline canvas)

Adding Sources

1. Click Add source on the canvas
2. Select source type (e.g., Azure Event Hubs)
3. Configure connection:
4. Event Hub namespace: iot-eventhub-namespace
5. Event Hub name: sensor-readings
6. Consumer group: $Default
7. Authentication: Connection string or Managed Identity
8. Events start flowing immediately

Adding Destinations

1. Click Add destination on the canvas
2. Select Eventhouse (or Lakehouse, Custom App, etc.)
3. Select your Eventhouse → KQL Database → table
4. Map the event fields to table columns
5. Events route automatically from source → destination

Transformations in Eventstream

Apply transformations to events in-flight without code:

Transformation	What It Does	Example
Filter	Keep only matching events	temperature > 100 (alert-worthy only)
Manage fields	Add, remove, rename columns	Add processed_at = NOW()
Group by	Aggregate over time windows	Average temperature per device every 5 minutes
Union	Combine multiple sources	Merge sensor-A and sensor-B streams
Expand	Flatten nested JSON arrays	Extract individual items from an array field

Source: IoT Hub → [Filter: temp > 100] → [Add: alert_level = "HIGH"] → Eventhouse
                → [Group by: device_id, 5min window, AVG(temp)] → Lakehouse

Events can be split to MULTIPLE destinations simultaneously — raw events to Eventhouse for real-time querying, aggregated events to Lakehouse for batch analysis.

Eventhouse: Storing Event Data

What Is an Eventhouse

An Eventhouse is a storage container optimized for time-series and event data. Inside it, you create one or more KQL Databases, each containing tables designed for fast querying over massive volumes (petabytes of data, billions of rows).

Eventhouse: production_eventhouse
  ├── KQL Database: iot_analytics
  │     ├── Table: sensor_readings (100M rows/day)
  │     ├── Table: device_metadata (10K rows)
  │     └── Table: alerts (50K rows/day)
  │
  └── KQL Database: web_analytics
        ├── Table: clickstream (500M rows/day)
        ├── Table: page_views (200M rows/day)
        └── Table: sessions (50M rows/day)

Creating an Eventhouse

1. + New item → Eventhouse
2. Name: analytics_eventhouse
3. A KQL Database is automatically created inside it
4. Name the database: iot_analytics

Key Characteristics

• Optimized for time-series: data is automatically indexed by ingestion time
• Massive scale: handles petabytes with sub-second query performance
• Hot/warm/cold caching: frequently accessed data stays in fast cache
• Automatic compression: typically 10-15x compression on raw data
• Retention policies: auto-delete data older than X days

KQL Database: Querying Events

What Is KQL (Kusto Query Language)

KQL is the query language for Eventhouse data. It is similar to SQL but optimized for log and time-series analysis — pipe-based syntax (like Unix commands) instead of SQL’s clause-based syntax.

KQL vs SQL Comparison

SQL	KQL
SELECT * FROM sensors WHERE temp > 100	sensors \| where temp > 100
SELECT device, AVG(temp) FROM sensors GROUP BY device	sensors \| summarize avg(temp) by device
SELECT TOP 10 * FROM sensors ORDER BY timestamp DESC	sensors \| top 10 by timestamp desc
SELECT COUNT(*) FROM sensors WHERE timestamp > '2026-05-25'	sensors \| where timestamp > datetime(2026-05-25) \| count
SELECT device, temp FROM sensors WHERE temp BETWEEN 50 AND 100	sensors \| where temp between (50 .. 100) \| project device, temp

Key difference: KQL uses a pipe (|) to chain operations from left to right. SQL uses clauses (SELECT, FROM, WHERE, GROUP BY). KQL reads like a Unix pipeline — data flows through transformations.

Essential KQL Queries

Filtering and Projecting

// Get recent sensor readings
sensor_readings
| where timestamp > ago(1h)          // Last 1 hour
| where device_id == "sensor-042"
| project timestamp, temperature, humidity, pressure   // Select columns
| order by timestamp desc
| take 100                            // Limit to 100 rows

// Text search (contains, startswith, matches regex)
application_logs
| where message contains "ERROR"
| where service startswith "payment"
| where timestamp > ago(30m)

Aggregation and Time-Based Analysis

// Average temperature per device in the last hour
sensor_readings
| where timestamp > ago(1h)
| summarize avg_temp = avg(temperature),
            max_temp = max(temperature),
            reading_count = count()
    by device_id
| order by avg_temp desc

// Time buckets: average temperature every 5 minutes
sensor_readings
| where timestamp > ago(24h)
| summarize avg_temp = avg(temperature)
    by bin(timestamp, 5m)             // 5-minute buckets
| render timechart                     // Visualize as time chart

// Top 10 pages by view count today
page_views
| where timestamp > startofday(now())
| summarize views = count() by page_url
| top 10 by views

Time Series Analysis

// Detect trends: is temperature rising or falling?
sensor_readings
| where timestamp > ago(7d)
| make-series avg_temp = avg(temperature) on timestamp step 1h by device_id
| extend trend = series_fit_line(avg_temp)
| project device_id, trend_slope = toreal(trend[0])
| where abs(trend_slope) > 0.5        // Significant trend only

// Compare this week vs last week
sensor_readings
| where timestamp > ago(14d)
| summarize avg_temp = avg(temperature)
    by bin(timestamp, 1d)
| extend week = iff(timestamp > ago(7d), "This Week", "Last Week")

Anomaly Detection (Built-in)

// Automatic anomaly detection on time series
sensor_readings
| where timestamp > ago(7d)
| make-series temp = avg(temperature) on timestamp step 1h
| extend anomalies = series_decompose_anomalies(temp)
| mv-expand timestamp, temp, anomalies
| where anomalies != 0                // Show only anomalous points

Real-Time Dashboards

Creating a Real-Time Dashboard

1. + New item → Real-Time Dashboard
2. Name: IoT Monitoring Dashboard
3. Click + Add tile
4. Write a KQL query → select visualization type → add to dashboard

Tiles and Visualizations

Visualization	Best For
Time chart	Trends over time (temperature, traffic)
Bar chart	Comparisons (events per device, errors per service)
Stat/Card	Single numbers (total events today, current average)
Map	Geographic data (device locations, regional traffic)
Table	Detailed records (recent errors, latest transactions)
Heatmap	Density (events by hour-of-day × day-of-week)

Auto-Refresh

Dashboards auto-refresh at configurable intervals: – Every 30 seconds (near real-time monitoring) – Every 1 minute (standard operational dashboard) – Every 5 minutes (analytical dashboard)

Data Activator (Alerts and Actions)

Data Activator (Reflex) monitors your streaming data and triggers actions when conditions are met:

Setting Up Alerts

Condition: sensor_readings | where temperature > 100
Action: Send email to ops@company.com
Frequency: Every time condition is met (with 5-minute cooldown)

Condition: error_events | summarize errors = count() by bin(timestamp, 5m) | where errors > 50
Action: Post to Teams channel #incidents

Alert → Action Combinations

Trigger	Action
Temperature exceeds threshold	Email operations team
Error rate spikes above 50/min	Teams notification to on-call
No events received for 10 minutes	Email + Teams (system might be down)
Transaction amount > $10,000	Flag for fraud review

Real-World Scenario 1: IoT Factory Monitoring

Factory: 500 machines, each sending sensor readings every 5 seconds

Eventstream: ES_Factory_Sensors
  Source: Azure IoT Hub (500 devices × 12 readings/min = 6,000 events/min)
  Transform: Add machine_zone (from device metadata lookup)
  Destinations:
    → Eventhouse (raw readings for KQL analysis)
    → Lakehouse (5-minute aggregates for historical analysis)

KQL Dashboard: Factory Operations
  Tile 1: Real-time temperature heatmap by zone
  Tile 2: Machines exceeding threshold (red = critical)
  Tile 3: Average vibration by machine (last 1 hour)
  Tile 4: Anomaly detection chart (auto-detected outliers)
  Auto-refresh: every 30 seconds

Data Activator Alert:
  IF temperature > 150°C for any machine → Teams alert to maintenance team
  IF vibration anomaly detected → email to engineering lead

Real-World Scenario 2: E-Commerce Clickstream Analytics

Website: 50,000 concurrent users, 2M clicks/hour

Eventstream: ES_Clickstream
  Source: Azure Event Hubs (from website tracking SDK)
  Fields: user_id, page_url, timestamp, device, geo_location, session_id

KQL Dashboard: Product Analytics (Real-Time)
  Tile 1: Active users right now (count distinct user_id last 5 min)
  Tile 2: Top 10 pages by views (live, updating every 30 sec)
  Tile 3: Conversion funnel (product view → cart → checkout → purchase)
  Tile 4: Geographic heatmap of current traffic
  Tile 5: Bounce rate by landing page (last 1 hour)

Business Impact:
  Marketing launches a campaign at 2 PM → dashboard shows traffic spike at 2:01 PM
  Product page error detected → clickstream shows drop-off at specific page
  A/B test results visible in real-time, not next-day

Real-World Scenario 3: Financial Transaction Monitoring

Bank: 100,000 transactions/hour

Eventstream: ES_Transactions
  Source: Kafka (from core banking system)
  Transform: Enrich with customer risk profile (lookup)
  Destinations:
    → Eventhouse (all transactions for KQL querying)
    → Reflex (alert on suspicious patterns)

KQL Queries:
  // Unusual velocity: >5 transactions in 10 minutes for one account
  transactions
  | where timestamp > ago(10m)
  | summarize txn_count = count(), total_amount = sum(amount) by account_id
  | where txn_count > 5

  // Geographic anomaly: transaction from a different country than usual
  transactions
  | where country != customer_home_country
  | where amount > 1000

Data Activator:
  IF same card used in 2 different countries within 1 hour → block card + alert
  IF single transaction > $10,000 → flag for manual review

Real-World Scenario 4: Application Log Analytics

Microservices: 20 services generating 50,000 log lines/minute

Eventstream: ES_App_Logs
  Source: Event Hubs (from centralized logging)
  Fields: timestamp, service, level (INFO/WARN/ERROR), message, trace_id

KQL Dashboard: DevOps Monitoring
  Tile 1: Error rate by service (last 1 hour, line chart)
  Tile 2: Top 10 error messages (table, grouped by count)
  Tile 3: P99 response time by service (time chart)
  Tile 4: Error spike detection (anomaly detection)

KQL Query: Trace a specific request across all services
  app_logs
  | where trace_id == "abc-123-def-456"
  | order by timestamp asc
  | project timestamp, service, level, message
  // Shows the full journey of one request through 20 services

Batch + Real-Time: The Complete Architecture

REAL-TIME PATH (hot):
  Event Hubs → Eventstream → Eventhouse → KQL Dashboard (seconds latency)
  Use for: monitoring, alerts, real-time decisions

BATCH PATH (warm/cold):
  Event Hubs → Eventstream → Lakehouse → Notebook → Warehouse → Power BI
  Use for: historical analysis, aggregations, ML training

COMBINED:
  Same Eventstream routes to BOTH Eventhouse AND Lakehouse simultaneously.
  Real-time dashboards show NOW. Power BI reports show trends over weeks/months.

  ┌─────────────┐    ┌────────────┐    ┌───────────┐    ┌─────────────┐
  │ Event Hubs  │───►│ Eventstream │───►│ Eventhouse │───►│ KQL Dashboard│
  │             │    │            │    └───────────┘    └─────────────┘
  │             │    │            │
  │             │    │            │───►┌───────────┐    ┌──────────┐    ┌──────────┐
  └─────────────┘    └────────────┘    │ Lakehouse │───►│ Notebook │───►│ Power BI │
                                       └───────────┘    └──────────┘    └──────────┘

Real-Time Intelligence vs Azure Stream Analytics vs Event Hubs

Feature	Fabric Real-Time Intelligence	Azure Stream Analytics	Azure Event Hubs
Purpose	Full RTI workload (ingest + store + query + visualize)	Stream processing (SQL-like queries on streams)	Message ingestion only (no processing)
Query language	KQL	Stream Analytics SQL	None (just passes messages)
Storage	Eventhouse (built-in, petabyte-scale)	No built-in storage	Short-term retention only
Visualization	Real-Time Dashboards (built-in)	Power BI (separate)	None
Alerting	Data Activator (built-in)	Separate (Logic Apps)	Separate
Part of Fabric	✅ Yes	❌ Separate Azure service	❌ Separate (but integrates as source)
Best for	End-to-end streaming analytics in Fabric	Complex stream processing outside Fabric	Raw event ingestion

Common Mistakes

1. Using batch pipelines for real-time needs — if you need data freshness under 5 minutes, use Real-Time Intelligence, not scheduled pipelines. Pipelines are for batch.

2. Sending everything to Eventhouse — raw, unfiltered events waste storage. Use Eventstream transformations to filter and aggregate before storing.

3. Not setting retention policies — without retention, your Eventhouse grows forever. Set policies (e.g., 90 days) to auto-delete old data.

4. Writing SQL instead of KQL — KQL is pipe-based, not clause-based. sensor_readings | where temp > 100 not SELECT * FROM sensor_readings WHERE temp > 100. Learn the KQL syntax.

5. Not using the dual-path architecture — real-time data should flow to BOTH Eventhouse (real-time queries) AND Lakehouse (historical batch analysis). One Eventstream, two destinations.

6. Dashboard refresh too frequent — 30-second refresh on a dashboard nobody watches wastes capacity. Match refresh frequency to actual monitoring needs.

Interview Questions

Q: What is Real-Time Intelligence in Fabric? A: Fabric’s streaming analytics workload consisting of four components: Eventstream (ingest and route streaming data), Eventhouse (store at petabyte scale), KQL Database (query with Kusto Query Language), and Real-Time Dashboards (visualize with auto-refresh). It handles continuous data flows like IoT sensors, clickstreams, and transaction logs.

Q: What is KQL and how does it differ from SQL? A: KQL (Kusto Query Language) is optimized for time-series and log data analysis. It uses pipe-based syntax where data flows left-to-right through transformations. SQL uses clause-based syntax (SELECT, FROM, WHERE). KQL has built-in time-series functions, anomaly detection, and rendering commands that SQL lacks. Both achieve similar results with different syntax.

Q: How do you build a real-time plus batch architecture in Fabric? A: Use one Eventstream with two destinations: route raw events to an Eventhouse for real-time KQL queries and dashboards, and simultaneously route aggregated events to a Lakehouse for historical batch analysis with notebooks and Power BI. This dual-path approach covers both real-time monitoring and long-term trend analysis.

Wrapping Up

Real-Time Intelligence completes Fabric’s data platform — batch (pipelines, notebooks, dataflows) for historical analysis, and streaming (Eventstream, Eventhouse, KQL, dashboards) for real-time monitoring. Most production implementations need both: the dual-path architecture where one Eventstream feeds both paths simultaneously.

Related posts: – Fabric Data Factory & Pipelines – Fabric Lakehouse Practical Guide – How Companies Receive Data – Mirrored Databases

---

Naveen Vuppula is a Senior Data Engineering Consultant and app developer based in Ontario, Canada. He writes about Python, SQL, AWS, Azure, and everything data engineering at DriveDataScience.com.