← Learning Hub Reference Databricks latency tiers Verified against docs: Aug–Sep 2026

Latency Tiers & Real-Time Fit

Quick-reference for deciding how fresh a Databricks pipeline needs to be, what that costs, and when to keep the workload out of the lakehouse entirely.

1. The tier table

Tier	Latency	Build with	Compute profile	Use when
Incremental / triggered	Minutes	Serverless job + `Trigger.AvailableNow()`, or triggered Lakeflow pipeline	Starts in seconds, runs backlog, exits	Medallion ETL, scheduled forecasts, market-deadline outputs
Continuous	Seconds	Lakeflow pipeline in continuous mode	Always-running cluster	Live monitoring, imbalance views, sub-minute freshness
Real-time mode	~300 ms typical, <1 s tail, 5 ms floor	`.trigger(realTime='5 minutes')`	Dedicated compute, all stages scheduled at once	Fraud detection, operational alerting, immediate automated action

Sources: Real-time mode concepts, Triggered vs. continuous pipeline mode, Streaming on serverless compute.

The seam at "a few minutes"

Docs place triggered mode at "every 10 minutes, hourly, or daily" and continuous at "every 10 seconds and a few minutes". A 5-minute cadence sits on the boundary — decide by benchmarking cost per day on your real volume, not by reading the doc harder.

2. Trigger syntax

# Triggered / incremental — cheapest, minute-level freshness
(df.writeStream
   .trigger(availableNow=True)
   .option("checkpointLocation", checkpoint_path)
   .toTable("catalog.schema.target"))

# Time-based micro-batch — classic compute only, NOT serverless
.trigger(processingTime="5 minutes")

# Real-time mode — sub-second, dedicated compute
.trigger(realTime="5 minutes")   # arg = batch length, not latency

# Deprecated since DBR 11.3 — migrate to availableNow
.trigger(once=True)

⚠ Serverless trigger constraints

Supported: Trigger.AvailableNow() (recommended), Trigger.Once() (deprecated).
Not supported: Trigger.ProcessingTime(interval), Trigger.Continuous(interval).
No explicit trigger → query fails with INFINITE_STREAMING_TRIGGER_NOT_SUPPORTED (Spark defaults to ProcessingTime("0 seconds")).
Always-on stream on serverless → use a Lakeflow pipeline in continuous mode.
Serverless run cap: 7 days. Longer runs are terminated and not retried.

3. Production rules for streaming jobs

Rule	Why
Never use all-purpose compute for streaming	Use jobs compute; new resources deploy on retry
Never enable classic autoscaling on a streaming job	Scale-down is unreliable; use Lakeflow enhanced autoscaling
One checkpoint directory per query, in cloud storage	Checkpoint reuse corrupts progress tracking
Prefer Lakeflow pipelines over hand-rolled streaming	Databricks' explicit recommendation for new pipelines
Make `foreachBatch` logic idempotent	`foreachBatch` is at-least-once, not exactly-once
Never leave the trigger unset	Default is 0 ms polling → runaway cloud storage API charges
Don't call `awaitTermination()` in a job	Blocks the Jobs service from tracking the query
Configure jobs to auto-restart on failure	Continuous job trigger retries with exponential backoff

Source: Production considerations for Structured Streaming.

4. Levers if you must go faster (in order of cost)

Shrink the fast lane. Move everything that does not depend on fresh data onto a slower schedule. Free.
Output mode update instead of append where supported; never complete. Requires an idempotent sink.
Shorten the watermark. Less state, lower latency — at the cost of dropping late records. Start around 2× your latency SLA.
Remove storage hops. Splitting logic across jobs adds a round trip to object storage each time.
RocksDB state store + changelog checkpointing for stateful queries; async state checkpointing if checkpointing dominates batch duration.
Async progress tracking for continuous streams (incompatible with AvailableNow/Once).
Low-latency source and sink. Kafka / Kinesis / Pub-Sub / Delta CDF in, message bus or operational DB out.
Real-time mode. Last resort. Dedicated compute, less frequent checkpointing, longer replay on failure.

5. Out-of-scope test

Keep it out of the lakehouse if…

A machine acts on the output with no human or market step in between.
It is safety, protection, or control logic (trip, curtailment command, setpoint).
It terminates an industrial protocol (OPC UA, IEC 60870-5-104, Modbus).
It needs a per-request response under ~50 ms.
It holds a socket open, listens on a port, or answers HTTP — that is an app, not a pipeline.

6. Energy-domain sizing heuristics

Signal	Typical cadence	Lane
NWP run (ICON, ECMWF, GFS)	1–6 hours	Slow
SCADA telemetry	1–10 minutes	Fast
Meter / settlement data	15 min – daily	Slow
Intraday market prices	Continuous, gate-driven	Fast
Model retraining	Daily – weekly	Slow

Rule: the fast lane recomputes only what fresh telemetry can change — measured output, availability, curtailment state, and a short-horizon bias/persistence correction. Everything else rides the slow lane.

← Lesson 12: Databricks for 5-Minute SCADA Architecture Cheatsheet →

Rechtliches: Impressum