Configuring Lineage#

Since Cosmos 1.1, it uses internally the openlineage-integration-common to parse artefacts generated with dbt commands and create lineage events.

To emit lineage events, Cosmos can use one of the following:

  1. Airflow official OpenLineage provider, or

  2. Additional libraries.

No change to the user DAG files is required to use OpenLineage.

Known limitations#

This feature is only available for the local and virtualenv execution methods (read execution modes for more information).

Additionally, since Cosmos uses the open-source openlineage-integration-common, it relies on this library to support specific dbt adapters. As of 27 December 2024, the version 1.26.0 of this package supports:

  • Athena

  • BigQuery

  • Databricks

  • DuckDB

  • Dremio

  • Postgres

  • Redshift

  • Snowflake

  • Spark

  • SQLServer

Contributions are also welcome in the OpenLineage project to support more adaptors.

Installation#

If using Airflow 2.7 or higher, install apache-airflow-providers-openlineage.

Otherwise, install Cosmos using astronomer-cosmos[openlineage].

Configuration#

If using Airflow 2.7, follow the instructions on how to configure OpenLineage.

Otherwise, follow these instructions.

Namespace#

Cosmos will use the Airflow [openlineage] namespace property as a namespace, if available.

Otherwise, it attempts to use the environment variable OPENLINEAGE_NAMESPACE as the namespace.

Finally, if neither are defined, it uses "cosmos" as the namespace.