> ## Documentation Index
> Fetch the complete documentation index at: https://openmetadata-format-2-0-connector-overview-pages.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Usage Workflow Guide | OpenMetadata Ingestion Workflows

> Monitor usage patterns from queries and dashboards with workflow-based usage extraction.

export const connector_0 = "bigquery"

# Usage Workflow

Learn how to configure the Usage workflow from the UI to ingest Query history data from your data sources.

This workflow is available ONLY for the following connectors:

* [BigQuery](/v2.0.x-SNAPSHOT/connectors/database/bigquery)
* [Clickhouse](/v2.0.x-SNAPSHOT/connectors/database/clickhouse)
* [Databricks](/v2.0.x-SNAPSHOT/connectors/database/databricks)
* [Exasol](/v2.0.x-SNAPSHOT/connectors/database/exasol)
* [MSSQL](/v2.0.x-SNAPSHOT/connectors/database/mssql)
* [PostgreSQL](/v2.0.x-SNAPSHOT/connectors/database/postgres)
* [Redshift](/v2.0.x-SNAPSHOT/connectors/database/redshift)
* [Snowflake](/v2.0.x-SNAPSHOT/connectors/database/snowflake)

If your database service is not yet supported, you can use this same workflow by providing a Query Log file!

Learn how to do so 👇

<CardGroup cols={1}>
  <Card title="Usage Workflow through Query Logs" href="/v2.0.x-SNAPSHOT/connectors/ingestion/workflows/usage/usage-workflow-query-logs">
    Configure the usage workflow by providing a Query Log file.
  </Card>
</CardGroup>

## UI Configuration

Once the metadata ingestion runs correctly and we are able to explore the service Entities, we can add Query Usage information.

This will populate the **Queries** tab from the Table Entity Page.

<img src="https://mintcdn.com/openmetadata-format-2-0-connector-overview-pages/84XyyDHxYM3pZsNc/public/images/features/ingestion/workflows/usage/query-tab.png?fit=max&auto=format&n=84XyyDHxYM3pZsNc&q=85&s=fccffcef59edeebc64bfbf90ed4d35d7" alt="table-entity-page" width="2738" height="1348" data-path="public/images/features/ingestion/workflows/usage/query-tab.png" />

We can create a workflow that will obtain the query log and table creation information from the underlying database and feed it to OpenMetadata. The Usage Agent will be in charge of obtaining this data.

### Step 1: Add a Usage Agent

1. Navigate to **Settings** > **Services** and select the service type.

2. Click the service you have added.

3. Open the **Agents** tab and click **Add Agent** > **Usage**.

   <img src="https://mintcdn.com/openmetadata-format-2-0-connector-overview-pages/84XyyDHxYM3pZsNc/public/images/features/ingestion/workflows/usage/add-usage-agent.png?fit=max&auto=format&n=84XyyDHxYM3pZsNc&q=85&s=644ff92f98176848db73176538a9169c" alt="Add usage agent" width="2820" height="1368" data-path="public/images/features/ingestion/workflows/usage/add-usage-agent.png" />

4. Give the agent a recognisable name and click **Next**.

### Step 2: Configure the Agent

Configure the core parameters for the Usage agent:

<img src="https://mintcdn.com/openmetadata-format-2-0-connector-overview-pages/84XyyDHxYM3pZsNc/public/images/features/ingestion/workflows/usage/usage-agent-setup.png?fit=max&auto=format&n=84XyyDHxYM3pZsNc&q=85&s=b3b272bfa521db03479ffb4bcdc76b08" alt="Usage Agent Setup" width="1910" height="804" data-path="public/images/features/ingestion/workflows/usage/usage-agent-setup.png" />

| **Field**               | **Description**                                                                                                    |
| :---------------------- | :----------------------------------------------------------------------------------------------------------------- |
| **Query Log Duration**  | How many days back to look in query logs to process usage data. Default is `1`.                                    |
| **Stage File Location** | Absolute file path for the temporary file used to store query logs before processing. Default is `/tmp/query_log`. |
| **Filter Condition**    | Optional condition to filter which queries are included from the query history.                                    |
| **Result Limit**        | Maximum number of query log results to process at a time. Default is `1000`.                                       |
| **Query Log File Path** | File path to the query logs, if reading from a file rather than querying the source directly.                      |
| **Process Query Cost**  | Toggle to enable processing of query cost data alongside usage.                                                    |

### Step 3: Schedule and Deploy

Choose how the usage agent should run:

<img src="https://mintcdn.com/openmetadata-format-2-0-connector-overview-pages/84XyyDHxYM3pZsNc/public/images/features/ingestion/workflows/usage/usage-schedule-ingestion.png?fit=max&auto=format&n=84XyyDHxYM3pZsNc&q=85&s=2899fc37d4773b94da2e8b0e6dc241df" alt="Schedule Usage Ingestion" width="1954" height="1456" data-path="public/images/features/ingestion/workflows/usage/usage-schedule-ingestion.png" />

* **Schedule**: Runs repeatedly based on a defined interval. Set the **Frequency** (Hourly, Daily, Weekly, Monthly, or Custom) and the **Time** to run.
* **On Demand**: Runs once when manually triggered.

Configure the remaining options:

* **Number of Retries**: How many times to retry if the ingestion fails.
* **Raise on Error**: Toggle on to surface errors immediately rather than silently skipping them.

Click **Add & Deploy** to save and activate the usage agent.

## YAML Configuration

In the [connectors](/v2.0.x-SNAPSHOT/connectors) section we showcase how to run the metadata ingestion from a JSON/YAML file using the Airflow SDK or the CLI via metadata ingest. Running a usage workflow is also possible using a JSON/YAML configuration file.

This is a good option if you wish to execute your workflow via the Airflow SDK or using the CLI; if you use the CLI a usage workflow can be triggered with the command `metadata usage -c FILENAME.yaml`. The `serviceConnection` config will be specific to your connector (you can find more information in the [connectors](/v2.0.x-SNAPSHOT/connectors) section), though the sourceConfig for the usage will be similar across all connectors.

## Query Usage

The Query Usage workflow will be using the `query-parser` processor.

After running a Metadata Ingestion workflow, we can run Query Usage workflow.
While the `serviceName` will be the same to that was used in Metadata Ingestion, so the ingestion bot can get the `serviceConnection` details from the server.

### 1. Define the YAML Config

This is a sample config for {connector_0} Usage:

<CodePreview>
  <ContentPanel>
    <ContentSection id={1} title="Source Configuration" lines="4">
      Configure the source type and service name for your usage workflow.
    </ContentSection>

    <ContentSection id={2} title="Usage Config Type" lines="6">
      **type**: Set to `DatabaseUsage` for database usage ingestion.
    </ContentSection>

    <ContentSection id={3} title="Query Log Duration" lines="8">
      **queryLogDuration**: Configuration to tune how far we want to look back in query logs to process usage data (in days).
    </ContentSection>

    <ContentSection id={4} title="Stage File Location" lines="10">
      **stageFileLocation**: Temporary file name to store the query logs before processing. Absolute file path required.

      Note that the location is a directory that will be cleaned at the end of the ingestion.
    </ContentSection>

    <ContentSection id={5} title="Result Limit" lines="11">
      **resultLimit**: Configuration to set the limit for query logs.
    </ContentSection>

    <ContentSection id={6} title="Query Log File Path" lines="13">
      **queryLogFilePath**: Configuration to set the file path for query logs. If instead of getting the query logs from the database we want to pass a file with the queries.
    </ContentSection>

    <ContentSection id={7} title="Processor Configuration" lines="14-16">
      Choose the `query-parser` processor to parse and process the query logs.
    </ContentSection>

    <ContentSection id={8} title="Stage Configuration" lines="17-20">
      Configure the staging location for table usage data before it's sent to OpenMetadata.
    </ContentSection>

    <ContentSection id={9} title="Bulk Sink Configuration" lines="21-24">
      Configure the bulk sink for metadata usage ingestion.
    </ContentSection>
  </ContentPanel>

  <CodePanel fileName="{connector}_usage.yaml">
    ```yaml theme={null}
    source:
      type: bigquery-usage
      serviceName: bigquery
      sourceConfig:
        config:
          type: DatabaseUsage
          # Number of days to look back
          queryLogDuration: 7
          # This is a directory that will be DELETED after the usage runs
          stageFileLocation: <path to store the stage file>
          # resultLimit: 1000
          # If instead of getting the query logs from the database we want to pass a file with the queries
          # queryLogFilePath: path-to-file
    processor:
      type: query-parser
      config: {}
    stage:
      type: table-usage
      config:
        filename: /tmp/bigquery_usage
    bulkSink:
      type: metadata-usage
      config:
        filename: /tmp/bigquery_usage
    ```
  </CodePanel>
</CodePreview>

### 2. Run with the CLI

After saving the YAML config, we will run the command the same way we did for the metadata ingestion:

```bash theme={null}
metadata usage -c <path-to-yaml>
```