The Lakehouse: Mapping Databricks to Weather-to-Power
This lesson gives you the mental model you need to make architectural decisions on Databricks. By the end, you'll be able to sketch how weather data flows through the platform and becomes a power forecast — and name the Databricks components responsible for each stage.
Your Domain Model
Before diving into Databricks, let's pin down the domain you're modelling. Every architectural decision traces back to this:
Key Domain Concepts
Asset hierarchy: A wind park is an asset with a geo-location. It contains turbines (sub-assets). You need forecasts at both levels — individual turbine and aggregated park.
Spatial mapping: Weather models produce gridded data (e.g., 0.25° resolution). You map grid cells to asset locations to get site-specific weather.
Time series everywhere: Weather variables, power output, and telemetry are all time series — timestamped sequences keyed by asset or turbine.
Dual access: The same time series data must be queryable for ad-hoc analysis (data scientists exploring patterns) and flow through the production pipeline (scheduled forecasts feeding trading desks).
The Lakehouse Concept
Databricks is built on the lakehouse architecture — a single platform that combines the cheap, scalable storage of a data lake with the reliability and performance of a data warehouse. No more choosing between the two.[1]
Key Insight
The lakehouse eliminates the "two-system" problem. Weather data (large, semi-structured, high-frequency) and power forecasts (structured, query-optimised) live in the same platform, governed by the same catalog. Crucially, time series data stored once in Delta tables can be accessed by both production pipelines (Workflows/Jobs) and ad-hoc analysts (SQL Warehouses/Notebooks) — no duplication required.
Three properties make this possible:
Delta Lake — an open storage layer that adds ACID transactions, schema enforcement, and time travel to Parquet files on cloud object storage (ADLS, S3, GCS).[2]
Unity Catalog — a unified governance layer for data, ML models, and code. One place to define access control and track lineage.
Compute separation — storage and compute are decoupled. You scale clusters independently of your data volume.
The Medallion Architecture
The standard pattern for organising data in a lakehouse is the medallion architecture: three layers (Bronze → Silver → Gold) that progressively refine raw data into business-ready assets.[3]
Why this matters for your team
Layer
Your Data
Architectural Concern
Bronze
Raw NWP files (ECMWF, GFS), SCADA feeds per turbine, asset registry (parks + turbines + locations)
Ingestion reliability, schema-on-read, no data loss, preserve raw grid structure
Silver
Weather time series mapped to asset geo-locations, telemetry per turbine (aligned UTC), park-level aggregates
Geo-spatial join (grid → asset), time series alignment, asset hierarchy modelling, dual access (analysis + pipeline)
Gold
Power forecasts per turbine & per park, confidence intervals, model accuracy metrics
SLA, latency, granularity (turbine vs. park), serving to trading/scheduling systems
Databricks Components — The Decision Map
As Tech Lead, you pick from these building blocks. Here's how they map to weather-to-power:
Component
What It Does
Your Use Case
Delta Lake
ACID storage layer on object storage
All tables are Delta. Time travel reproduces any forecast. Concurrent reads (analysts) and writes (pipeline) are safe.
Unity Catalog
Governance: access control, lineage, discovery
Separate catalogs for dev/prod. Track which weather model + asset version produced which forecast. Asset hierarchy as managed tables.
Workflows (Jobs)
Orchestrate multi-task pipelines
Scheduled pipeline: ingest weather → geo-map to assets → engineer features → run model per park → publish forecast.
Delta Live Tables
Declarative ETL with built-in quality checks
Expectations on time series (e.g., "wind speed ≥ 0", "no gaps > 1h"). Quality gates before ML consumption.
MLflow
ML lifecycle: tracking, registry, serving
Track experiments per asset type. Register champion model. Serve per-turbine or per-park predictions.
Clusters
Managed Spark compute
Job clusters (ephemeral) for pipeline runs. Interactive clusters for data scientists exploring time series.
SQL Warehouses
Serverless SQL compute
Analysts query Silver/Gold time series. Dashboards for forecast accuracy per park. The "analysis" access path.
The End-to-End Picture
Here's how a single forecast cycle flows through Databricks:
Dual Access — The Architecture That Serves Both Paths
Silver-layer time series tables (weather per asset, telemetry per turbine) are the single source of truth. The production pipeline reads them on schedule to produce forecasts. Analysts query the same tables interactively via SQL Warehouses or notebooks. Delta Lake makes this safe: concurrent reads never block writes, and time travel lets analysts pin to a consistent snapshot without interfering with pipeline freshness.
Your First Architectural Decision
As Tech Lead, one of your earliest decisions is: Delta Live Tables (DLT) vs. custom Spark jobs orchestrated by Workflows?
Criterion
Delta Live Tables
Custom Spark + Workflows
Setup effort
Lower — declarative SQL/Python
Higher — imperative code
Data quality
Built-in expectations
You write validation logic
Flexibility
Constrained to DLT patterns
Full control
Cost visibility
Opaque (enhanced tier pricing)
Transparent cluster costs
Complex logic
Harder for custom file formats
Easy — plain Spark code
Recommendation for weather-to-power
Weather data arrives in specialised formats (GRIB2, NetCDF) that need custom parsing libraries (cfgrib, xarray). This pushes Bronze ingestion toward custom Spark jobs. Silver/Gold transformations — mostly SQL-expressible — can benefit from DLT's declarative quality checks. A hybrid approach is common.
Check Your Understanding
In the medallion architecture, where does raw GRIB weather data first land?
The Bronze layer as append-only raw records
The Silver layer after parsing and validation
The Gold layer as business-ready forecasts
The Feature Store for direct ML consumption
Correct. Bronze is the landing zone — raw, immutable, append-only. No transformation happens here. This preserves full fidelity for reprocessing.
Not quite. Bronze is always the first stop. Raw data lands unmodified so you can always reprocess from source if your transformation logic changes.
Which Databricks component tracks ML experiments and serves the champion power forecast model?
Delta Live Tables with quality expectations
Unity Catalog with governed model lineage
MLflow for tracking, registry, and serving
Workflows with multi-task job orchestration
Correct. MLflow handles the ML lifecycle end-to-end: experiment tracking (hyperparameters, metrics), model registry (versioning, stage promotion), and model serving (real-time or batch endpoints).
MLflow is the answer. While Unity Catalog does govern model access, it's MLflow that actually tracks experiments, registers model versions, and serves predictions.
Why might you choose custom Spark jobs over Delta Live Tables for Bronze weather ingestion?
Custom jobs have built-in data quality expectations
Delta Live Tables cannot write to Delta format tables
Weather files (GRIB2, NetCDF) need custom parsing libraries
Custom jobs are always cheaper than Delta Live Tables
Exactly. GRIB2 and NetCDF require specialised libraries (cfgrib, xarray, eccodes) that work best in imperative Spark/Python code. DLT excels at SQL-expressible transformations, not custom binary format parsing.
The key reason is format complexity. GRIB2 and NetCDF are domain-specific binary formats that require specialised parsing libraries — this doesn't fit DLT's declarative model well.
How does the lakehouse enable dual access — analysts querying time series while the production pipeline writes new data?
By copying time series into a separate analytics database nightly
By blocking pipeline writes during analyst query windows
Delta Lake's MVCC allows concurrent readers and writers on the same table
By routing analysts to Bronze and the pipeline to Gold only
Correct. Delta Lake uses multi-version concurrency control (MVCC). Readers see a consistent snapshot while writers append new data. No locks, no copies, no coordination needed. Analysts can even use time travel to pin to a specific version.
The answer is MVCC. Delta Lake's transaction log allows multiple concurrent readers and writers without interference. No data copying or scheduling workarounds needed.
Primary Source
📖 Recommended Reading
What is a data lakehouse? — Databricks Documentation. ~10 min read. This is the canonical explanation of how Delta Lake, Unity Catalog, and compute come together. Read sections 1–3 for the foundation this lesson builds on.
What's Next
Now that you have the architectural map, the next lesson will zoom into modelling assets and time series in Delta Lake — how to structure tables for the asset hierarchy (park → turbine), partition time series for performance, and enable the dual-access pattern.