← Learning Hub Lesson 11 Duration: ~16 min Topic: 5-min real-time forecast updates

Real-Time Forecasting with Databricks

⚠ Errata — read Lesson 12 alongside this

This lesson shows how to build a 5-minute pipeline but skips the prior question of whether you should, and its code assumes classic compute. On serverless, trigger(processingTime=…) is not supported and Trigger.Once() is deprecated. See Lesson 12 — Is Databricks Right for 5-Minute SCADA Data? for the corrected tier guidance and the current recommendation to build this as a Lakeflow pipeline.

In the energy market, new telemetry and market signals can arrive every 5 minutes. This lesson shows how Databricks supports a low-latency forecasting pipeline that ingests frequent updates, refreshes features, and writes a new forecast with a short SLA.

What "real-time" means here

For your use case, real-time means:

New asset telemetry or market data arriving every 5 minutes
A forecast that is updated shortly after each arrival
Low-latency production flow, while preserving Delta-based analytics access

Fast lane + slow lane

Use a fast lane for frequent 5-minute updates and a slow lane for daily model training. The fast lane ingests and scores fresh data; the slow lane retrains models on historical data. Both must share the same Delta-based asset model and data governance.

Architecture pattern

Ingestion: structured streaming to Bronze

This is the first important step. A micro-batch pipeline captures every 5-minute payload and writes it to a Delta Bronze table with checkpointing. That keeps the data durable, queryable, and replayable.

from pyspark.sql.functions import col

streaming_df = (
    spark.readStream
         .format("cloudFiles")
         .option("cloudFiles.format", "json")
         .option("cloudFiles.schemaLocation", "/tmp/checkpoints/telemetry_schema")
         .option("cloudFiles.inferColumnTypes", "true")
         .load("/mnt/data/realtime/telemetry/")
         .select(
             col("asset_id"),
             col("timestamp"),
             col("wind_speed"),
             col("power_output"),
             col("market_price")
         )
)

query = (
    streaming_df.writeStream
        .format("delta")
        .option("checkpointLocation", "/tmp/checkpoints/bronze_telemetry")
        .outputMode("append")
        .trigger(processingTime="5 minutes")
        .table("prod.bronze.telemetry")
)

Incremental Silver + scoring

Use a second micro-batch stage that reads the Bronze stream and updates Silver features and forecasts every 5 minutes. A practical pattern is foreachBatch(), which gives you deterministic batch processing plus Delta table writes.

def process_batch(batch_df, batch_id):
    # 1) Enrich telemetry with asset metadata
    enriched = (
        batch_df
        .join(spark.table("prod.silver.dim_assets"), "asset_id", "left")
        .withColumn("event_time", col("timestamp"))
    )

    # 2) Create incremental feature table rows
    features = compute_features(enriched)

    # 3) Score with MLflow model
    model = mlflow.pyfunc.load_model("models:/wind-forecast/latest")
    scored = model.predict(features.toPandas())

    # 4) Write Gold forecast output
    to_write = (
        features
        .withColumn("forecast", lit(scored))
        .withColumn("produced_at", current_timestamp())
    )

    (to_write.write
         .format("delta")
         .mode("append")
         .saveAsTable("prod.gold.forecast_ts"))

query = (
    spark.readStream
         .table("prod.bronze.telemetry")
         .writeStream
         .foreachBatch(process_batch)
         .option("checkpointLocation", "/tmp/checkpoints/silver_scoring")
         .trigger(processingTime="5 minutes")
         .start()
)

Note: In production, avoid converting the entire micro-batch to pandas unless the batch size is small. A distributed Spark scoring solution or MLflow model serving endpoint is safer for larger fleets.

Realtime readiness guidelines

Keep Bronze raw: preserve every 5-minute event so analysts can replay and audit the fast lane.
Use Delta for dual access: both analytics and production read the same tables via Delta, so the fast updates are queryable immediately.
Partition by asset + date: helps both low-latency writes and efficient time-series queries.
Model version in Gold: store model_version and produced_at for traceability.

Operational rule

For a 5-minute update cycle, design the pipeline so each run completes well under 5 minutes. That gives you headroom for retries and ensures the next cycle is not delayed.

When micro-batch is better than continuous streaming

For your forecast use case, a 5-minute micro-batch pipeline is usually the best fit:

It matches the natural update cadence of telemetry.
It simplifies feature computation and model scoring.
It keeps Delta semantics strong and makes debugging easier.

Continuous streaming is useful when latency must be seconds, but it adds complexity in state management, exactly-once joins, and operational overhead. For 5-minute forecast refreshes, micro-batch is the right tradeoff.

Key implementation patterns

Pattern	Why it matters	Where to use it
Auto Loader / cloudFiles	Efficient, schema-aware incremental file ingestion	Telemetry files landing in cloud storage
Delta Streaming + foreachBatch	Deterministic, repeatable batch processing for scoring	Fast lane Silver + Gold updates
MLflow model registry	Versioned models + safe stage promotion	Predict with the same model used in production
Checkpointing	Exactly-once delivery and restart safety	Every streaming write to Delta or model scoring job

Check Your Understanding

You receive telemetry every 5 minutes and update the forecast after each arrival. Which pattern preserves both analytics and production needs?

Write forecasts only to a separate file store outside Databricks
Use Delta Bronze stream for raw events and write forecasts to Gold Delta
Ingest raw events to Bronze Delta, compute Silver features, then append new forecasts to Gold Delta
Use pure Kafka streaming without Delta because Delta is too slow for 5-minute updates

Correct. The Delta Bronze → Silver → Gold pattern keeps raw events durable for analytics while also producing a production-ready forecast. This preserves the dual-access requirement.

The right answer is the Delta medallion pattern. Bronze preserves raw 5-minute events, Silver builds features, and Gold stores production forecasts. This maintains both analytics and production access.

Why is micro-batch processing usually better than continuous streaming for 5-minute forecast updates?

Micro-batch always uses fewer compute resources than streaming
Streaming cannot write to Delta tables reliably
Micro-batch matches the cadence, keeps Delta semantics, and reduces state complexity
Continuous streaming cannot include ML models in the pipeline

Right. For 5-minute updates, micro-batch is simpler and aligns with the arrival cadence. It gives you robust Delta writes and easier batch-based feature computation without introducing large streaming state complexity.

The main reason is cadence and engineering complexity. Micro-batch is often the right fit for 5-minute updates because it preserves Delta reliability and avoids long-lived streaming state. It is not because Delta is unreliable or models cannot be used.

Your 5-minute forecast pipeline must not miss a cycle. What should you monitor?

Whether the model registry has a new version every hour
Only the SQL Warehouse query latency for downstream users
Pipeline latency, Bronze write freshness, and checkpoint health
Whether Unity Catalog table comments are up to date

Correct. For a 5-minute update pipeline, the important signals are whether the pipeline completed within the SLA, whether Bronze is receiving fresh events, and whether streaming checkpoints are healthy.

The critical metrics are pipeline latency and streaming checkpoint health. Model registry versions and table comments matter for governance, but they do not ensure the 5-minute cycle is met.

Primary Source

📖 Recommended Reading

Databricks Structured Streaming documentation — covers micro-batch triggers, checkpointing, Delta writes, and production streaming patterns for low-latency pipelines.

What's Next

Now you know how to design the fast lane for 5-minute updates. Ask your teacher to build the next lesson around realtime model serving or streaming feature state in the same energy-market pipeline.

← Previous: Cost, Performance & Operations Next: Is Databricks Right for This? →

Need an example notebook for 5-minute streaming ingestion and scoring? Ask me to generate one.

Rechtliches: Impressum