← Learning Hub Lesson 5 Duration: ~14 min Topic: Governance & Access Control

Unity Catalog & Governance

This lesson teaches you how to organise your Databricks workspace so multiple teams (data engineers, data scientists, analysts, downstream consumers) can collaborate safely. By the end, you'll be able to design a catalog structure, assign permissions, and explain data lineage through your weather-to-power pipeline.

The Three-Level Namespace

Unity Catalog introduces a three-level naming convention for all data objects:^[1]

catalog.schema.table

-- Examples in your domain:
prod.bronze.weather_raw
prod.silver.weather_ts
prod.silver.dim_assets
prod.gold.forecast_ts
dev.silver.weather_ts        -- dev environment copy

Recommended catalog structure for weather-to-power

Catalog	Purpose	Who Writes	Who Reads
`prod`	Production pipeline tables (Bronze/Silver/Gold)	Service principals (pipeline jobs)	Everyone (read-only for humans)
`dev`	Development/staging — mirrors prod structure	Data engineers	Data engineers, data scientists
`sandbox`	Exploration — analysts create personal schemas	Analysts, data scientists	Owner only (or shared explicitly)

Key Principle: Prod is Machine-Written

No human writes directly to prod. Only service principals (attached to Workflow jobs) have write access. This prevents accidental corruption and ensures all prod data is traceable to a pipeline run. Humans read prod, experiment in sandbox, and promote changes through dev.

Permissions Model

Unity Catalog uses a hierarchical permission model. Grants cascade downward (catalog → schema → table). Deny at any level blocks access.^[2]

-- Grant patterns for your team

-- 1. Pipeline service principal: full access to prod
GRANT USE CATALOG ON CATALOG prod TO `pipeline-service-principal`;
GRANT USE SCHEMA ON SCHEMA prod.bronze TO `pipeline-service-principal`;
GRANT USE SCHEMA ON SCHEMA prod.silver TO `pipeline-service-principal`;
GRANT USE SCHEMA ON SCHEMA prod.gold TO `pipeline-service-principal`;
GRANT ALL PRIVILEGES ON SCHEMA prod.bronze TO `pipeline-service-principal`;
GRANT ALL PRIVILEGES ON SCHEMA prod.silver TO `pipeline-service-principal`;
GRANT ALL PRIVILEGES ON SCHEMA prod.gold TO `pipeline-service-principal`;

-- 2. Data scientists: read prod, write dev + sandbox
GRANT USE CATALOG ON CATALOG prod TO `data-scientists`;
GRANT SELECT ON SCHEMA prod.silver TO `data-scientists`;
GRANT SELECT ON SCHEMA prod.gold TO `data-scientists`;
GRANT ALL PRIVILEGES ON CATALOG dev TO `data-scientists`;
GRANT ALL PRIVILEGES ON CATALOG sandbox TO `data-scientists`;

-- 3. Analysts: read prod gold + silver, write sandbox
GRANT USE CATALOG ON CATALOG prod TO `analysts`;
GRANT SELECT ON SCHEMA prod.silver TO `analysts`;
GRANT SELECT ON SCHEMA prod.gold TO `analysts`;
GRANT ALL PRIVILEGES ON CATALOG sandbox TO `analysts`;

-- 4. Downstream systems: read gold only
GRANT USE CATALOG ON CATALOG prod TO `trading-system`;
GRANT USE SCHEMA ON SCHEMA prod.gold TO `trading-system`;
GRANT SELECT ON SCHEMA prod.gold TO `trading-system`;

Service principals vs. user accounts

Identity Type	Use For	Example
Service principal	Automated pipelines, external system access	`sp-weather-pipeline`, `sp-trading-reader`
Group	Team-level permissions (easier to manage)	`data-engineers`, `data-scientists`, `analysts`
User account	Interactive work (notebooks, SQL editor)	Individual team members

Data Lineage

Unity Catalog automatically tracks lineage — which tables read from which, and which jobs produced which outputs.^[3]

Lineage answers critical questions for your team:

Impact analysis: "If I change dim_assets schema, what breaks downstream?"
Root cause: "This forecast looks wrong — which Bronze data and model version produced it?"
Compliance: "Prove that the trading system only reads from validated Gold tables."

Catalog Design Decisions

Option A: Environment-based catalogs (recommended)

prod.bronze.weather_raw
prod.silver.weather_ts
dev.bronze.weather_raw
dev.silver.weather_ts

Pro: Clear separation. One GRANT gives full dev access. Prod is locked down by default.
Con: Queries must change catalog name between environments (mitigated by parameterisation).

Option B: Domain-based catalogs

weather.bronze.raw_forecasts
weather.silver.asset_ts
power.gold.forecasts

Pro: Natural for very large orgs with many domains.
Con: Environment separation requires per-schema grants. More complex for a single-team product.

Recommendation for Your Team

Use environment-based catalogs (prod/dev/sandbox). Your domain is coherent (weather-to-power is one product), and environment separation is the primary governance need. Within each catalog, use schemas for medallion layers (bronze/silver/gold). This keeps permissions simple and pipeline promotion predictable.

Managing dim_assets: The Master Data Problem

Your asset registry (parks, turbines, locations) is master data — it changes rarely but has outsized impact when it does. Unity Catalog helps manage this:

-- Table comment documents ownership and update process
COMMENT ON TABLE prod.silver.dim_assets IS 
  'Asset master data. Source of truth for all geo-locations and hierarchy.
   Owner: data-engineering team.
   Update process: PR-reviewed changes via dev.silver.dim_assets → promotion job.
   Breaking changes require downstream notification.';

-- Column comments for discoverability
COMMENT ON COLUMN prod.silver.dim_assets.parent_asset_id IS 
  'FK to parent asset (park). NULL if this row IS a park.';
COMMENT ON COLUMN prod.silver.dim_assets.hub_height_m IS 
  'Turbine hub height in metres. Used for wind speed extrapolation. NULL for parks.';

Change management for master data

Changes are made in dev.silver.dim_assets (engineers have write access)
Validated: run Silver pipeline in dev to verify joins still work
Promoted: a controlled job copies to prod.silver.dim_assets (using MERGE)
Lineage shows which downstream tables are affected

Tags and Classification

Unity Catalog supports tags for discoverability and governance policies:

-- Tag tables by data classification
ALTER TABLE prod.silver.weather_ts SET TAGS ('pii' = 'none', 'retention' = '2years');
ALTER TABLE prod.gold.forecast_ts SET TAGS ('pii' = 'none', 'sla' = 'hourly');
ALTER TABLE prod.silver.dim_assets SET TAGS ('data_class' = 'master_data');

-- Tag columns (useful for sensitive location data if needed)
ALTER TABLE prod.silver.dim_assets ALTER COLUMN latitude SET TAGS ('geo' = 'true');
ALTER TABLE prod.silver.dim_assets ALTER COLUMN longitude SET TAGS ('geo' = 'true');

Check Your Understanding

A data scientist wants to experiment with a new feature for the ML model. Where should they create tables?

prod.silver — so they can test against production time series data
dev.gold — since they're developing a new production output feature
sandbox catalog — personal schema for experiments that don't affect others
prod.gold — write experimental results directly for fast iteration speed

Correct. The sandbox catalog exists for exploratory work. Data scientists read from prod (Silver/Gold) but write experiments to their own sandbox schema. This prevents accidental interference with production or other team members' work.

Sandbox is correct. Scientists read from prod Silver/Gold for production data, but write experimental tables to their own sandbox schema. Prod is machine-written only; dev is for pipeline development, not ad-hoc experiments.

The trading system needs to read power forecasts but nothing else. What's the minimal permission set?

USE CATALOG on prod, USE SCHEMA on prod.gold, SELECT on prod.gold
USE CATALOG on prod, SELECT on all schemas in the prod catalog
ALL PRIVILEGES on prod.gold.forecast_ts table only without schema grants
USE CATALOG on prod, USE SCHEMA on prod.silver and prod.gold, SELECT on both

Correct. Unity Catalog requires USE CATALOG to even see the catalog, USE SCHEMA to access objects in a schema, then SELECT for read access. Granting SELECT on the schema level covers all tables in prod.gold. This is the minimal path — no Silver access, no write access.

The answer is USE CATALOG + USE SCHEMA + SELECT on prod.gold only. Each level is required for access to traverse the namespace. The trading system doesn't need Silver access — it should only see validated Gold outputs.

Your pipeline writes to prod.silver.weather_ts. What should own the write permission?

The data engineer's personal account who developed the pipeline code
The data-engineers group so any team member can run the pipeline manually
A service principal dedicated to the weather pipeline workflow job
A shared team account with credentials stored in the job configuration

Right. Service principals are non-human identities attached to automated jobs. They provide: audit trail (which SP wrote what), least-privilege (only pipeline SP has prod write), and independence from individual team members leaving or changing roles.

Service principal is correct. Automated pipelines should use dedicated service principals — not human accounts. This ensures the pipeline runs regardless of personnel changes, provides clean audit trails, and enforces that prod is never written by humans directly.

Primary Source

📖 Recommended Reading

Unity Catalog best practices — Databricks Documentation. Covers catalog organisation, permission patterns, and migration strategies. Also see Privilege types for the full permission reference.

What's Next

With governance in place, the next lesson covers Feature Engineering & the Feature Store — how to transform Silver time series into ML-ready features, manage feature tables, and enable reproducible model training.

← Previous: Bronze → Silver Next: Feature Engineering & Feature Store →

Questions about catalog structure, permission inheritance, or managing cross-team access? Ask your teacher.

Rechtliches: Impressum