Skip to main content

Connect Databricks Unity Catalog

Required Inputs​

Prepare:

  • Databricks workspace URL.
  • Databricks account ID, if account-level APIs are used.
  • Authentication method approved by your Databricks administrator.
  • Catalogs, schemas, and workspaces in scope.
  • Decision on whether notebook, job, and pipeline metadata can be included.

Scope The First Sync​

Start with a narrow scope:

  • One catalog.
  • One or two schemas.
  • A limited set of jobs or pipelines.
  • One destination Purview collection.

Expand after the first validation succeeds.

Metadata Collected​

Depending on configuration and permissions, Lineage Advisor may collect:

  • Catalog, schema, table, view, and column names.
  • Unity Catalog lineage relationships where available.
  • Job, task, notebook, and pipeline identifiers.
  • Runtime and operational metadata needed to explain lineage.
  • Owners, tags, descriptions, and classifications where available and enabled.

Lineage Advisor does not need table data values to publish lineage metadata.

Validation Checks​

The connector validates:

  • Workspace endpoint connectivity.
  • Authentication.
  • Catalog and schema visibility.
  • Lineage table or API availability.
  • Minimum metadata read permissions.

Operational Guidance​

  • Use a dedicated service principal or token.
  • Avoid personal access tokens for production.
  • Rotate secrets under your enterprise policy.
  • Keep initial catalog scope small until lineage quality is verified.