Lesson 12 — Is Databricks Right for 5-Minute SCADA Data?

You have 5-minute telemetry arriving from a SCADA system and you need a solar and wind production forecast out the other end. The question you asked is the right one for a Tech Lead to ask before the team builds: is Databricks the correct tool for this, or are we about to force a batch analytics platform to do a job it was not designed for?

This lesson gives you a defensible answer, the evidence behind it, and — more importantly — the reasoning frame you can reuse when the next latency question arrives.

The short answer

Step 1: Separate the two numbers everyone conflates

"5-minute data" is a statement about arrival cadence. It says nothing about your latency requirement. These are independent, and mixing them up is the single most common source of over-engineered energy data platforms.

Question	Your answer	What it drives
How often does data arrive?	Every 5 minutes, per asset	Trigger cadence, batch size, file layout
How stale may the output be?	Set by the market/dispatch deadline	Compute tier, cost, architecture

For a production forecast, the deadline is almost never "as fast as physically possible". It is a calendar: intraday gate closure, the schedule you must submit to the TSO, the redispatch request, the hour-ahead nomination. If your forecast needs to be on someone's desk before a gate that closes 15 minutes before delivery, your end-to-end budget is minutes — and minutes is exactly the tier Databricks is cheapest and simplest at.

Step 2: Know the three latency tiers Databricks actually offers

Databricks does not have one streaming answer. It has three, with sharply different cost and complexity profiles. Knowing which one you are buying is most of the architecture decision.

Step 3: Recognise the two clocks in your pipeline

Tier	Latency	How you build it	Compute
Incremental / triggered	Minutes	Serverless job on a schedule with `Trigger.AvailableNow()`, or a triggered Lakeflow pipeline	Starts, runs the batch, exits — cheapest
Continuous	Seconds	Lakeflow pipeline in continuous mode	Always-on cluster — more expensive
Real-time mode	Sub-second	`.trigger(realTime='5 minutes')` in Structured Streaming or Lakeflow	Dedicated compute, held open

This is the insight that makes the cost work for a weather-to-power forecast. Your system has two independent refresh rates, and running both at the faster one is pure waste.

Numerical weather prediction updates a handful of times a day. Your SCADA feed updates 288 times a day. The 5-minute path should do only what fresh telemetry can actually change: measured output, availability and curtailment state, and a short-horizon bias/persistence correction on top of the standing NWP-driven forecast. That is a small, cheap, incremental job — which is precisely why the triggered tier fits.

Step 4: Know where Databricks is the wrong answer

Being able to say "no" credibly is what makes your "yes" worth something to stakeholders. Four things should not go through the lakehouse:

Step 5: The recommended shape

Workload	Fit	Where it belongs
Turbine trip logic, protection, curtailment commands	No	SCADA/PLC. Safety and control loops never route through analytics.
Protocol termination (OPC UA, IEC 60870-5-104, Modbus)	No	An edge gateway or historian. Databricks reads what they publish.
Sub-50 ms per-request lookup for a web app or API	No	A serving store or model serving endpoint fed from Gold — not a Delta scan.
One Delta write per SCADA record as it arrives	No	Batch at the gateway or message bus. Each write is a transaction and a file.
5-min ingest, QC, feature build, forecast, history, analysis	Yes	Exactly what the lakehouse is for.

Build it as a serverless Lakeflow pipeline. Databricks explicitly recommends Lakeflow pipelines over hand-rolled Structured Streaming for new pipelines, because it removes the production infrastructure management. [Production considerations for Structured Streaming]

⚠ Serverless constraint that will bite you

On serverless compute, only Trigger.AvailableNow() and the deprecated Trigger.Once() are supported. Trigger.ProcessingTime(interval) and Trigger.Continuous(interval) are not, and a query with no explicit trigger fails with INFINITE_STREAMING_TRIGGER_NOT_SUPPORTED. For an always-on stream on serverless, use a Lakeflow pipeline in continuous mode instead. [Serverless compute limitations]

Decide it yourself

Work the tree below for your own workload. Try it twice: once for the forecast pipeline, once for something you might be tempted to bolt on — like a curtailment alarm.

1. From data arriving to the result being usable downstream, what is your hard deadline?

Minutes — a market gate, dispatch schedule, or reporting deadline
Seconds — live operational monitoring or an imbalance view
Milliseconds — an automated control or protection decision

2. Does anything act on the result automatically, with no human or market step in between?

No — a trader, scheduler, or downstream system consumes it
Yes — equipment or a control system acts on it directly

3. If one run is 10 minutes late, what happens?

A stale schedule or trade gets submitted — real money
Nothing much — the next run catches up

Databricks — triggered tier

The cheapest correct answer. Serverless job or triggered Lakeflow pipeline, Trigger.AvailableNow(), scheduled every 5–15 minutes. Compute starts in seconds, processes the backlog since the last checkpoint, and exits. Resist anything more elaborate until a measurement forces you.

Databricks — triggered tier, with an SLA

Same build, but freshness is now a production concern. Size the job so a run finishes well inside the interval, alert on Bronze staleness and on run duration, and make the downstream consumer read produced_at so it can refuse to trade on a stale forecast. If runs regularly overrun, move to a continuous Lakeflow pipeline before you move to real-time mode.

Databricks — continuous tier

Lakeflow pipeline in continuous mode on serverless. You are buying an always-running cluster, so justify the seconds: name the decision that gets worse at 5 minutes. If you cannot name one, you are at the triggered tier and paying for the continuous one.

Not a lakehouse workload

Keep this in SCADA, the PLC, or a dedicated stream processor at the edge. Databricks real-time mode can hit ~300 ms, but a control or protection decision should not depend on a cloud analytics platform's availability at all. Let Databricks observe, record, and analyse the outcome — not make the call.

Check Your Understanding

What should set the latency requirement for your SCADA-fed forecast pipeline?

The arrival cadence of the SCADA feed
The market deadline the forecast must meet
The maximum write throughput of Delta tables
The refresh interval of the weather model

Correct. Arrival cadence tells you how often to trigger; the business deadline tells you how fresh the output must be. Deriving the SLA from the deadline is what keeps you off the expensive tiers you do not need.

The deadline drives it. Data arriving every 5 minutes does not mean the answer is needed within 5 minutes — the gate closure, schedule submission, or dispatch decision defines the real budget, and that budget picks your compute tier.

Which configuration gives minute-level freshness at the lowest cost on serverless?

Real-time mode running on dedicated always-on compute
Serverless job with Trigger.AvailableNow on a schedule
All-purpose compute kept warm between the manual runs
Continuous trigger with a zero-second processing interval

Correct, and it is the documented recommendation: a scheduled Trigger.AvailableNow() job is the most cost-efficient streaming pattern on serverless when minute-level latency is acceptable — compute starts in seconds, runs the batch, shuts down.

It is the scheduled Trigger.AvailableNow() job. Real-time mode needs dedicated compute, all-purpose compute is explicitly discouraged for streaming, and time-based or continuous triggers are not even supported on serverless.

Which of these workloads should stay outside Databricks entirely?

Turbine curtailment alarms driving immediate control action
Nightly retraining of the site power curve model
Ad-hoc analysis of past forecast error by asset
Five-minute aggregation of raw turbine telemetry rows

Correct. Anything a machine acts on without a human or market in between belongs in SCADA or the PLC. The lakehouse can record and analyse the alarm afterwards, but the decision must not depend on cloud availability.

The control action is the odd one out. Retraining, error analysis, and 5-minute aggregation are all analytical workloads the lakehouse handles well — but a curtailment command acting directly on equipment must never route through an analytics platform.

Why not write each SCADA record into Delta the moment it arrives?

Delta cannot accept appends from any streaming source
Every write commits a transaction and a small file
Unity Catalog blocks writes above an hourly frequency
Streaming tables require a fixed five-minute batch schedule

Correct. Delta's ACID guarantee is per commit, so per-record writes mean per-record transactions and a flood of tiny files. Batch at the gateway or message bus and let one micro-batch produce one commit.

It is the commit and small-file cost. Delta happily accepts streaming appends and Unity Catalog imposes no frequency limit — the problem is that one write equals one transaction plus one file, which destroys read performance and inflates maintenance.

Primary Source

📖 Recommended Reading

Streaming on serverless compute — Databricks. This is the single most useful page for your question: it is an explicit decision guide mapping streaming use cases to the right product, and it states plainly which triggers are and are not available. Read the "Choose the right streaming configuration" table and the "Workloads that are not streaming pipelines" section.

Follow it with Real-time mode concepts to see exactly what you would be buying at the top tier — and why the docs steer medallion workloads away from it.