| Layer | Daten | Partition | Zentrale Eigenschaft |
|---|---|---|---|
| Bronze | Rohes GRIB2 → Parquet, SCADA-Telemetrie, Asset-Register | init_time | Unveränderlich, nur Append |
| Silver | Wetter-Zeitreihe je Asset, Leistungs-Zeitreihe je Turbine, dim_assets | asset_id + ZORDER(timestamp) | Doppelter Zugriff (Pipeline + Analyst) |
| Gold | Prognose je Turbine + Park, Konfidenzintervalle, Genauigkeit | asset_id | Vertrag für nachgelagerte Systeme |
asset_id STRING PK
asset_type STRING (wind_park|turbine)
parent_asset_id STRING FK (NULL for parks)
latitude DOUBLE
longitude DOUBLE
hub_height_m DOUBLE
capacity_kw DOUBLE
metadata MAP<STRING,STRING>asset_id STRING PK (partition)
timestamp TIMESTAMP PK
forecast_ref_time TIMESTAMP
wind_speed_ms DOUBLE
wind_speed_hub_ms DOUBLE
wind_direction_deg DOUBLE
temperature_c DOUBLE
irradiance_wm2 DOUBLE
pressure_hpa DOUBLE
source_model STRINGasset_id STRING PK (partition)
timestamp TIMESTAMP PK
power_kw DOUBLE
rotor_speed_rpm DOUBLE
nacelle_temp_c DOUBLE
availability DOUBLE
status_code STRINGasset_id STRING (partition)
asset_type STRING
timestamp TIMESTAMP
forecast_power_kw DOUBLE
confidence_low_kw DOUBLE
confidence_high_kw DOUBLE
capacity_factor DOUBLE
forecast_ref_time TIMESTAMP
model_version STRING
produced_at TIMESTAMP
source_model STRINGWeather API (ECMWF/GFS/ICON)
→ [Download GRIB2] → bronze.weather_raw (PARTITION BY init_time)
→ [Geo-join + derive] → silver.weather_ts (PARTITION BY asset_id, ZORDER timestamp)
→ [Compute features] → features.wind_power_features (PK: asset_id, timestamp)
→ [ML inference @champion] → gold.forecast_ts
→ [Serve] → Trading (JDBC) / Grid (Delta Sharing) / Dashboards (SQL Warehouse)| Größe | Formel |
|---|---|
| Windgeschwindigkeit | $\sqrt{u^2 + v^2}$ |
| Windrichtung | $\text{atan2}(-u, -v) \times \frac{180}{\pi} \mod 360$ |
| Wind auf Nabenhöhe | $v_{10} \times (h_{hub} / 10)^{\alpha}$ (α ≈ 0,14 offen, 0,10 offshore) |
| Temperatur | $T_C = T_K - 273.15$ |
| Einstrahlung (momentan) | $(ssrd[t] - ssrd[t{-}1]) / \Delta t_{sec}$ |
Metastore
├── prod (nur maschinell beschrieben)
│ ├── bronze → weather_raw
│ ├── silver → weather_ts, power_ts, dim_assets
│ ├── gold → forecast_ts, forecast_accuracy
│ └── features → wind_power_features
├── dev (hier entwickeln die Engineers)
│ ├── bronze / silver / gold (Spiegel von prod)
└── sandbox (Analysten + Data Scientists explorieren)
└── <user_schema>| Principal | prod | dev | sandbox |
|---|---|---|---|
| Pipeline-SP | ALL (bronze/silver/gold) | — | — |
| Data Engineers | SELECT (alle) | ALL | ALL |
| Data Scientists | SELECT (silver/gold) | ALL | ALL |
| Analysten | SELECT (silver/gold) | — | ALL |
| Nachgelagerte Systeme | SELECT (nur gold) | — | — |
| Aufgabe | Typ | Worker | VM |
|---|---|---|---|
| GRIB-Ingestion | Job-Cluster | 0 (Single-Node) | Memory-optimiert (E4s) |
| Silver-Transformationen | Job-Cluster | 2–4 | General Purpose (D4s) |
| ML-Inferenz | Job-Cluster | 1 | Memory-optimiert (E8s), ML-Runtime |
| Analysten-Abfragen | SQL Warehouse | Serverless | Automatisch (skaliert auf null) |
| Aufgabe | Häufigkeit | Befehl |
|---|---|---|
| OPTIMIZE + ZORDER | Wöchentlich | OPTIMIZE table ZORDER BY (timestamp) |
| VACUUM | Wöchentlich | VACUUM table RETAIN N DAYS |
| Bronze-Aufbewahrung | Monatlich | Partitionen > 90 Tage löschen |
| Modell-Neutraining | Monatlich (oder bei Drift) | Neu trainieren → validieren → promoten |
| Entscheidung | Empfehlung | Lektion |
|---|---|---|
| DLT vs. eigenes Spark für Bronze | Eigenes (GRIB2 braucht cfgrib) | 1 |
| Modell der Asset-Hierarchie | Flache Tabelle + Parent-Zeiger | 2 |
| Partitionsstrategie für Zeitreihen | BY asset_id, ZORDER timestamp | 2 |
| Ansatz für den Geo-Join | Nächster Nachbar (auf Gitter einrasten) | 4 |
| Versionierung der Prognosen | Alle speichern, neueste ausliefern (View) | 4 |
| Catalog-Struktur | Umgebungsbasiert (prod/dev/sandbox) | 5 |
| Globales vs. Modell je Asset | Global + Asset-Features (zum Start) | 7 |
| Konfidenzintervalle | Quantilsregression (LightGBM) | 8 |
| IaC-Werkzeug | Databricks Asset Bundles | 9 |
Rechtliches: Impressum