Cost, Performance & Operations

This final lesson covers the operational realities: controlling cloud spend, optimising Delta table performance, sizing clusters efficiently, and establishing runbooks for your production weather-to-power platform. As Tech Lead, cost accountability and operational health are your responsibility.

Cost Model — Where the Money Goes

Databricks cost has two components: compute (DBUs) and cloud infrastructure (VMs + storage).^[1]

Estimated monthly cost (illustrative)

These are rough estimates for a moderate-scale fleet (~50 turbines, 10 parks, Europe region). Actual costs depend on cloud provider, DBU tier (Premium vs. Standard), and data volumes.

Delta Table Optimisation

Delta tables accumulate small files over time (from frequent appends). Unchecked, this degrades read performance.

OPTIMIZE — compact small files

-- Run weekly (or let auto-optimize handle it)
OPTIMIZE prod.silver.weather_ts;
OPTIMIZE prod.silver.power_ts;
OPTIMIZE prod.gold.forecast_ts;

-- With Z-ORDER for time-range query performance
OPTIMIZE prod.silver.weather_ts ZORDER BY (timestamp);
OPTIMIZE prod.silver.power_ts ZORDER BY (timestamp);

VACUUM — delete old file versions

-- Remove files older than retention period (default 7 days)
-- After VACUUM, time travel to those versions is no longer possible
VACUUM prod.bronze.weather_raw RETAIN 30 DAYS;
VACUUM prod.silver.weather_ts RETAIN 14 DAYS;
VACUUM prod.gold.forecast_ts RETAIN 7 DAYS;

Auto-optimize (preferred for production)

-- Enable at table creation (already in our schemas):
TBLPROPERTIES (
  'delta.autoOptimize.optimizeWrite' = 'true',  -- write larger files
  'delta.autoOptimize.autoCompact'   = 'true'   -- compact small files
);

-- These reduce the need for manual OPTIMIZE, but Z-ORDER still needs 
-- periodic explicit OPTIMIZE runs (auto-compact doesn't Z-ORDER).

Cluster Sizing Guide

Data Retention Policy

-- Bronze: keep raw weather for reprocessing (30–90 days)
-- After that, Silver is the source of truth
ALTER TABLE prod.bronze.weather_raw
SET TBLPROPERTIES ('delta.deletedFileRetentionDuration' = '30 days');

-- Silver: keep indefinitely (primary analytical asset)
-- VACUUM removes only file versions, not logical data

-- Gold: keep forecast history for accuracy tracking (1–2 years)
-- Older forecasts can be archived to cold storage if needed

Operational Runbook

Incident: Pipeline failure (ingestion)

1. Check: Which source failed? (Workflow run UI → failed task)
2. Diagnose: API timeout? Auth failure? Source downtime? (Check logs)
3. Act: If transient → manual retry. If source down → wait, verify freshness SLA.
4. Communicate: If SLA breached → notify downstream (trading desk uses stale forecast).

Incident: Model accuracy degradation

1. Detect: forecast_accuracy table shows MAE rising over 7 days.
2. Diagnose: Seasonal shift? Asset change (new turbine)? Weather model update upstream?
3. Act: Retrain model on recent data. Validate challenger. Promote if improved.
4. Prevent: Schedule automated retraining (monthly or on drift detection).

Incident: Slow pipeline (duration alert)

1. Check: Which task is slow? (Workflow timeline view)
2. Diagnose: Data volume spike? Cluster undersized? Small files (needs OPTIMIZE)?
3. Act: Run OPTIMIZE if small-file issue. Scale cluster if data grew. Add partition pruning if missing.

Monitoring Dashboard

Key metrics to track on an operational dashboard (Databricks SQL Dashboard or external):

Check Your Understanding

Primary Source

Course Summary

You've now covered the complete Databricks weather-to-power architecture:

1. Lakehouse architecture — the mental model
2. Asset hierarchy & time series — table design
3. Weather ingestion — GRIB2, APIs, Bronze
4. Bronze → Silver — geo-join, derivation, versioning
5. Unity Catalog — governance and access control
6. Feature engineering — Feature Store, point-in-time
7. ML pipeline — MLflow, model registry, inference
8. Gold layer — serving downstream, confidence intervals
9. Workflows — orchestration, scheduling, alerting
10. Cost & Operations — this lesson

You've completed the foundation course. Ask your teacher to dive deeper into any topic, create hands-on exercises, or explore advanced topics (streaming, real-time inference, multi-model ensembles).

Rechtliches: Impressum