Metadata Ingestion: Incremental Extraction — Unity Catalog
This page describes how OpenMetadata implements incremental extraction for Unity Catalog, including the required permissions and the queries used to detect changed and deleted tables.Approach
To implement incremental extraction for Unity Catalog, OpenMetadata queries two system tables at the start of incremental processing for each catalog, using a watermark derived from the previous successful run timestamp:- Changed tables:
information_schema.tablesis filtered bylast_alteredto find tables modified since the last successful run. Each changed table is then fetched individually via the Databricks SDK. - Deleted tables:
system.access.auditis queried fordeleteTableevents scoped to the configured catalog. If thesystem.accessschema is unavailable, delete detection is skipped with a warning and only changed-table detection runs.
information_schema only lists currently existing tables.
Incremental extraction only runs for catalogs whose name matches
[A-Za-z0-9_]+. Catalogs with names outside this pattern (for example,
quoted names like cat-a-log) are skipped with a warning: no changed or
deleted tables are returned, and no exception is raised.Prerequisites
Before enabling incremental extraction, review the following visibility and permission requirements for the Unity Catalog system tables used at runtime.Changed table detection
No explicit grant is required forinformation_schema.tables: Databricks
automatically scopes its results to objects the querying principal can
already see through existing Unity Catalog privileges. Ensure the ingestion
service principal has standard read privileges (USE CATALOG, USE SCHEMA,
SELECT) on the tables you want incrementally tracked.
Deleted table detection (optional)
To detect dropped tables, the ingestion user also needs access tosystem.access.audit:
If these grants are absent, delete detection is skipped automatically and a
warning is logged.
Queries
The following queries run when OpenMetadata prepares the changed and deleted table maps for incremental processing. OpenMetadata replaces{start_timestamp} with the millisecond epoch watermark derived from the
previous successful run timestamp (for example, 1718000000000).
Changed tables
OpenMetadata filtersinformation_schema.tables by the last_altered column
to identify tables that were created or modified within the ingestion window.
Deleted tables
OpenMetadata queries the Unity Catalog audit log fordeleteTable events to
identify tables that were dropped within the ingestion window.